Upgrade your rooted Google Play Edition

Have you ever been annoyed by accidentally touching the touchpad on your laptop when you are typing?

I am using a Linux laptop with both trackpoint and touchpad so the touchpad is just an annoying redundant thing. Here is the one-liner to disable it:

xinput list | grep TouchPad | grep -Po 'id=\K([0-9]+)' | xargs xinput disable

This script performs the build, test and automatic checking of a Go package and its sub-packages using:

1. gofmt
2. goimports
3. golint
4. go vet
5. ineffassign
6. race detector
7. test coverage on package and its sub-packages, /vendor directories excluded
8. goveralls
9. gocyclo
10. misspell

Migrated from my Gist.

Dependencies

To setup all the dependencies need to run the script do:

$ go get -v github.com/client9/misspell/cmd/misspell
$ go get -v github.com/fzipp/gocyclo
$ go get -v github.com/golang/lint/golint
$ go get -v github.com/gordonklaus/ineffassign
$ go get -v github.com/h12w/gosweep
$ go get -v github.com/mattn/goveralls
$ go get -v golang.org/x/tools/cmd/goimports

Environment variables

• GOCYCLO_COMPLEXITY: maximum allowed function complexity threshold (default: 5).
• MISSPELL_LOCALE: English locale (default: US).

If you wish to set this for a project you can create a .gosweep file. For example:

$ cat .gosweep
GOCYCLO_COMPLEXITY=5
MISSPELL_LOCALE='US'

Continuous Integration

travis-ci

Example of .travis.yml file for Go:

language: go
cache:
  directories:
    - ${GOPATH}/src/github.com/${TRAVIS_REPO_SLUG}/vendor
    - ${GOPATH}/src/github.com/fzipp
    - ${GOPATH}/src/github.com/golang
    - ${GOPATH}/src/github.com/gordonklaus
    - ${GOPATH}/src/github.com/h12w
    - ${GOPATH}/src/github.com/mattn
go:
  - tip
  - 1.8
  - 1.7
  - 1.6
sudo: false

env:
    CI_SERVICE=travis-ci

install:
  - go get -v github.com/client9/misspell/cmd/misspell
  - go get -v github.com/fzipp/gocyclo
  - go get -v github.com/golang/lint/golint
  - go get -v github.com/gordonklaus/ineffassign
  - go get -v github.com/h12w/gosweep
  - go get -v github.com/mattn/goveralls
  - go get -v -u golang.org/x/tools/cmd/goimports

script:
  - bash ${GOPATH}/src/github.com/h12w/gosweep/gosweep.sh

Getgo is a concurrent, simple and extensible web scraping framework written in Go.

Quick start

Get Getgo

go get -u github.com/h12w/getgo

Define a task

This example is under the examples/goblog directory. To use Getgo to scrap structured data from a web page, just define the structured data as a Go struct (golangBlogEntry), and define a corresponding task (golangBlogIndexTask).

type golangBlogEntry struct {
	Title string
	URL   string
	Tags  *string
}

type golangBlogIndexTask struct {
	// Variables in task URL, e.g. page number
}

func (t golangBlogIndexTask) Request() *http.Request {
	return getReq(`http://blog.golang.org/index`)
}

func (t golangBlogIndexTask) Handle(root *query.Node, s getgo.Storer) (err error) {
	root.Div(_Id("content")).Children(_Class("blogtitle")).For(func(item *query.Node) {
		title := item.Ahref().Text()
		url := item.Ahref().Href()
		tags := item.Span(_Class("tags")).Text()
		if url != nil && title != nil {
			store(&golangBlogEntry{Title: *title, URL: *url, Tags: tags}, s, &err)
		}
	})
	return
}

Run the task

Use util.Run to run the task and print all the result to standard output.

	util.Run(golangBlogIndexTask{})

To store the parsed result to a database, a storage backend satisfying getgo.Tx interface should be provided to the getgo.Run method.

Understand Getgo

A getgo.Task is an interface to represent an HTTP crawler task that provides an HTTP request and a method to handle the HTTP response.

type Task interface {
	Requester
	Handle(resp *http.Response) error
}

type Requester interface {
	Request() *http.Request
}

A getgo.Runner is responsible to run a getgo.Task. There are two concrete runners provided: SequentialRunner and ConcurrentRunner.

type Runner interface {
	Run(task Task) error // Run runs a task
	Close()              // Close closes the runner
}

A task that stores data into a storage backend should satisfy getgo.StorableTask interface.

type StorableTask interface {
	Requester
	Handle(resp *http.Response, s Storer) error
}

A storage backend is simply an object satisfying getgo.Tx interface.

type Storer interface {
	Store(v interface{}) error
}

type Tx interface {
	Storer
	Commit() error
	Rollback() error
}

See getgo.Run method to understand how a StorableTask is combined with a storage backend and adapted to become a normal Task to allow a Runner to run it.

There are currently a PostgreSQL storage backend provided by Getgo, and it is not hard to support more backends (See getgo/db package for details).

The easier way to define a task for an HTML page is to define a task satisfying getgo.HTMLTask rather than getgo.Task, there are adapters to convert internally an HTMLTask to a Task so that a Runner can run an HTMLTask. The Handle method of HTMLTask provides an already parsed HTML DOM object (by html-query package).

type HTMLTask interface {
	Requester
	Handle(root *query.Node, s Storer) error
}

Similarly, a task for retrieving a JSON page should satisfy getgo.TextTask interface. An io.Reader is provided to be decoded by the encoding/json package.

type TextTask interface {
	Requester
	Handle(r io.Reader, s Storer) error
}

GSpec is an expressive, reliable, concurrent and extensible Go test framework that makes it productive to organize and verify the mind model of software.

• Expressive: a complete runnable specification can be organized via both BDD and table driven styles.
• Reliable: the implementation has minimal footprint and is tested with 100% coverage.
• Concurrent: test cases can be executed concurrently or sequentially.
• Extensible: customizable BDD cue words, expectations and test reporters.
• Compatible: “go test” is sufficient but not mandatory to run GSpec tests.

Quick start

Get GSpec

go get -u -f h12.io/gspec
go test h12.io/gspec/...

Write tests with GSpec

According to the convention of Go, write GSpec tests in file xxx_test.go to test code in xxx.go.

import (
	"fmt"

	"h12.io/gspec"
)

// Only one suite.Add is needed for each xxx_test.go file.
var _ = gspec.Add(func(s gspec.S) {
	// BDD cue word is customizible.
	describe, given, when, it := s.Alias("describe"), s.Alias("given"), s.Alias("when"), s.Alias("it")
	// expectation cue word is customizible too.
	expect := gspec.Alias(s.FailNow)

	// A BDD example.
	describe("an integer i", func() {
		// setup
		i := 2
		defer func() {
			// teardown (if any)
		}()
		given("another integer j", func() {
			j := 3
			when("j is added to i", func() {
				i += j
				it("should become the sum of original i and j", func() {
					expect(i).Equal(5) // a passing case
				})
			})
			when("j is minused from i", func() {
				i -= j
				it("should become the difference of j minus i", func() {
					expect(i).Equal(4) // a failing case
				})
			})
			when("j is multiplied to i", nil) // a pending case
		})
	})

	// A table-driven example.
	testcase := s.Alias("testcase")
	describe("integer summation", func() {
		for _, c := range []struct{ i, j, sum int }{
			{1, 2, 3}, // a passing case
			{1, 1, 0}, // a failing case
		} {
			testcase(fmt.Sprintf(`%d + %d = %d`, c.i, c.j, c.sum), func() {
				expect(c.i + c.j).Equal(c.sum)
			})
		}
	})
})

Write the following go test function for only once in any test file within the package (e.g. all_test.go).

import (
	"testing"
	"h12.io/gspec"
)

// Defined only once within a package.
func TestAll(t *testing.T) {
	gspec.Test(t)
}

Run tests with “go test”

Run all the tests concurrently (sequencially by default) and display errors.

go test -concurrent

Run all the tests and view the complete specification.

go test -v

Run only a failing test case (even it is an entry in the driven table):

go test -focus 1/1

html-query is a Go package that provides a fluent and functional interface for querying HTML DOM. It is based on go.net/html.

Examples

1. A simple example (under “examples” directory)

   r := get(`http://blog.golang.org/index`)
   defer r.Close()
   root, err := query.Parse(r)
   checkError(err)
   root.Div(Id("content")).Children(Class("blogtitle")).For(func(item *query.Node) {
       href := item.Ahref().Href()
       date := item.Span(Class("date")).Text()
       tags := item.Span(Class("tags")).Text()
       // ......
   })
   

2. Generator of html-query (under “gen” directory)

A large part of html-query is automatically generated from HTML spec. The spec is in HTML format, so the generator parses it using html-query itself.

Design

Here is a simple explanation of the design of html-query.

Functional query expressions

All functional definitions are defined in html-query/expr package.

1. Checker and checker composition
   A checker is a function that accept and conditionally returns a *html.Node.

   type Checker func(*html.Node) *html.Node
   

   Here are some checker examples:

   Id("id1")
   Class("c1")
   Div
   Abbr
   H1
   H2
   

   Checkers can be combined as boolean expressions:

   And(Id("id1"), Class("c1"))
   Or(Class("c1"), Class("c2"))
   And(Class("c1"), Not(Class("c2")))
   

2. Checker builder
   A checker builder is a function that returns a checker. “Id”, “Class”, “And”, “Or”, “Not” shown above are all checker builders. There are also some checker builder builder (function that returns a checker builder) defined in html-query when needed.

Fluent interface

Fluent interface (http://en.wikipedia.org/wiki/Fluent_interface) are defined in html-query package.

1. Root node
   Function Parse returns the root node of an html document.

2. Node finder
   Method Node.Find implements a BFS search for a node, e.g.

   node.Find(Div, Class("id1"))
   

   But usually you can write the short form:

   node.Div(Class("id1"))
   

3. Attribute getter Method Node.Attr can be used to get the value (or a regular expression submatch of the value) of a node, e.g.

   node.Attr("Id")
   node.Attr("href", "\(.*)")
   

   But usually you can write the short form:

   node.Id()
   node.Href("\(.*)")
   

4. Node iterator
   Method Node.Children and Node.Descendants each returns a node iterator (NodeIter). Method NodeIter.For can be used to loop through these nodes.

Alternative

If you prefer a jquery like DSL rather than functional way, you might want to try goquery.

Cwrap is a Go wrapper generator for C libraries.

Features

• No Cgo types exposed out of the wrapper package, and uses as less allocation/copy as possible.
• C name prefix mapped to Go packages, and a wrapper package can import another wrapper package.
• Follows Go naming conventions.
• C union.
• Use Go language features when possible:
  • string and bool.
  • Multiple return values.
  • Slice, slice of slice and slice of string.
  • struct with methods.
  • Go closures as callbacks.
• Stay out of the way when you need to do it manually for specified declarations.

Usage

Cwrap itself is a Go package rather than an executable program. Just fill a cwrap.Package struct literal and call its Wrap method to generate your wrapper package under $GOPATH. Here is a simple example:

Say you want to generate a wrapper package for SDL2, and its header is at

/usr/local/include/SDL2/SDL2.h

So the cwrap.Package literal looks like:

var sdl = &Package{
	PacName: "sdl",
	PacPath: "go-sdl",
	From: Header{
		Dir:           "/usr/local/include/",
		File:          "SDL2/SDL.h",
		OtherCode:     "#define _SDL_main_h",
		NamePattern:   `\ASDL(.*)`,
		Excluded:      []string{},
		CgoDirectives: []string{"pkg-config: sdl2"},
		BoolTypes:     []string{"SDL_bool"},
	},
	Included: []*Package{},
}

Then just call

err := sdl.Wrap()

Examples

In the examples directory, there are C libraries that I have successfully applied Cwrap, including:

• Cairo
• GSL (GNU Scientific Library)
• MuPDF
• PLplot
• SDL2 (Simple DirectMedia Layer)

You are very welcome to submit examples you think useful to others.

Applications

• gr: A minimal PDF viewer based on SDL2 and MuPDF (https://github.com/h12w/gr)

Issue Report

Cwrap may not cover every possible case and fails to come up with a corrresonding Go type or convertion, then the generated code may not be able to compile. When this happens, do the following steps:

1. Comment out the failed function wrappers till it compiles.
2. Add the C names of these failed functions to the excluded list (Package.From.Excluded).
3. Submit the generator example to me. I cannot guarantee anything but I will try to fix critical issues.

Limitations

• C variadic functions (…) are not supported.

Acknowledgement

Cwrap uses gccxml (http://gccxml.github.io) to parse C headers to an XML file. Thanks very much for their excellent work.