"Phil Dawes' Stuff"[1] title "Phil Dawes' Stuff" "Phil Dawes' Stuff"[1] link http://www.phildawes.net/blog "Phil Dawes' Stuff"[1] description "Semantic Web, Programming, stuff like that." "Phil Dawes' Stuff"[1] pubDate "Sat, 20 Oct 2007 15:18:21 +0000" "Phil Dawes' Stuff"[1] generator http://wordpress.org/?v=2.1.1 "Phil Dawes' Stuff"[1] language en "Frequent code checkpointing with git"[1] title "Frequent code checkpointing with git" "Frequent code checkpointing with git"[1] link http://www.phildawes.net/blog/2007/10/20/frequent-code-checkpointing-with-git/ "Frequent code checkpointing with git"[1] comments http://www.phildawes.net/blog/2007/10/20/frequent-code-checkpointing-with-git/#comments "Frequent code checkpointing with git"[1] pubDate "Sat, 20 Oct 2007 15:18:21 +0000"[1] "Frequent code checkpointing with git"[1] creator[1] "Phil Dawes" "Frequent code checkpointing with git"[1] category versioncontrol "Frequent code checkpointing with git"[1] category git "Frequent code checkpointing with git"[1] category dvcs http://www.phildawes.net/blog/2007/10/20/frequent-code-checkpointing-with-git/[1] isPermaLink false[255] "Frequent code checkpointing with git"[1] guid http://www.phildawes.net/blog/2007/10/20/frequent-code-checkpointing-with-git/[1] "A nice feature of distributed version control is that you can commit into your repository more often because you aren’t impacting others with each commit. Recently I’ve been taking this to the extreme using git and performing a commit almost at every save. I have an emacs key wired for the job: it save the [..." _xmlchars "A nice feature of distributed version control is that you can commit into your repository more often because you aren’t impacting others with each commit. Recently I’ve been taking this to the extreme using git and performing a commit almost at every save. I have an emacs key wired for the job: it save the [...]"[255] "Frequent code checkpointing with git"[1] description "A nice feature of distributed version control is that you can commit into your repository more often because you aren’t impacting others with each commit. Recently I’ve been taking this to the extreme using git and performing a commit almost at every save. I have an emacs key wired for the job: it save the [..." "

A nice feature of distributed version control is that you can commit into your repository more often because you aren’t impacting others with each commit. Recently I’ve been taking this to the extreme using git and performing a commit almost at every save. I have an emacs key wired for the job: it save the buffers and then runs ‘git commit -a -m “checkpoint”‘. During a coding session I hammer it frequently.

" _xmlchars "

A nice feature of distributed version control is that you can commit into your repository more often because you aren’t impacting others with each commit. Recently I’ve been taking this to the extreme using git and performing a commit almost at every save. I have an emacs key wired for the job: it save the buffers and then runs ‘git commit -a -m “checkpoint”‘. During a coding session I hammer it frequently.

I’ve found this approach particularly useful when I’m programming with unfamiliar tools (as I am at the moment with factor). I tend to re-write stuff a lot and occasionally change my mind mid-rewrite, wanting some old code back. Using git in the above manner makes it painless to checkpoint my work continuously and pull back changes from previous revisions when appropriate.

However there’s a problem when I want to share my repository with others: Nobody else is interested in the 100s of incremental checkpoint commits with no commit messages; they want to see commits in functional units which tally with the changelog. To make things more legable for them I need to be able to roll up the checkpoints into functional commits with full commit messages.

Originally I assumed that the best way to do this was to have two branches: one for checkpoint commits whilst developing, and a second public one for containing the larger functional commits. 

o--o--o--o--o--o--o--o--o  < --- checkpoint (dev) branch  /  ___/ ______/ /  /  __/|  |  |v  v  vo--o--o                    <--- public branch ('proper' commits)

I tried various approaches to merging multiple commits from the checkpoint branch into single commits in the public branch, but couldn&#8217;t find anything that worked.

I think the main problem with the 2 branch idea is that once you&#8217;ve rolled up the commits from one branch into a single commit in the other (e.g. with git merge &#8211;squash), they no longer match and so the branches don&#8217;t have a recent common ancestor. This means git is unable to track the histories by checksum and so each subsequent merge results in conflicts that must be resolved by hand.

I found the easiest way to get round this was to just copy the content over from one branch to the other rather than merging it. This could be done either by creating patches or by simply checking out the contents of one into the other with &#8216;git checkout branch .&#8217; (i.e. checkout branch &lt;path>). This removes the need to resolve conflicts, but the branches still don&#8217;t share common ancestors. Ultimately you have to manage the branches separately - for example you have to pull external changesets into each branch individually.

In the end the best way turned out to be to dispense with the 2nd public branch all together and just operate in one. My method is as follows:

I tag the most recent public commit in the branch, and then perform lots of checkpoint commits as I code. When I&#8217;m ready to roll up the checkpoint commits into the next &#8216;proper&#8217; commit I go back to the previous public commit with:

% git checkout public

Then, assuming master is the current branch I checkout the contents of the HEAD of the branch (i.e. all the checkpointed commits) into the working directory, but without moving the index:

% git checkout master .

Then I move the HEAD of the master branch to this point. I do this by deleting and recreating the branch again:

% git branch -D master% git checkout -b master

Finally I commit the changes and tag this as the new latest public commit:

- git commit -a- git tag -f public

And that&#8217;s it. Now I haven&#8217;t been using this technique for long, so there&#8217;s a good chance that something might trip me up in the future - If anybody can see a problem with this (or a better way) then I&#8217;d really appreciate a comment.

"[255] "Frequent code checkpointing with git"[1] encoded[1] "

A nice feature of distributed version control is that you can commit into your repository more often because you aren&#8217;t impacting others with each commit. Recently I&#8217;ve been taking this to the extreme using git and performing a commit almost at every save. I have an emacs key wired for the job: it save the buffers and then runs &#8216;git commit -a -m &#8220;checkpoint&#8221;&#8216;. During a coding session I hammer it frequently.

" "Frequent code checkpointing with git"[1] commentRSS[1] http://www.phildawes.net/blog/2007/10/20/frequent-code-checkpointing-with-git/feed/ "Frequent code checkpointing with git"[1] tag item "Phil Dawes' Stuff"[1] item "Frequent code checkpointing with git"[1] "More factor: tabular to triples"[1] title "More factor: tabular to triples" "More factor: tabular to triples"[1] link http://www.phildawes.net/blog/2007/10/11/more-factor-tabular-to-triples/ "More factor: tabular to triples"[1] comments http://www.phildawes.net/blog/2007/10/11/more-factor-tabular-to-triples/#comments "More factor: tabular to triples"[1] pubDate "Thu, 11 Oct 2007 20:40:13 +0000" "More factor: tabular to triples"[1] creator[1] "Phil Dawes"[1] "More factor: tabular to triples"[1] category General "More factor: tabular to triples"[1] category Programming "More factor: tabular to triples"[1] category factor http://www.phildawes.net/blog/2007/10/11/more-factor-tabular-to-triples/[1] isPermaLink false[255] "More factor: tabular to triples"[1] guid http://www.phildawes.net/blog/2007/10/11/more-factor-tabular-to-triples/[1] "I&#8217;ve been playing with Factor for a couple of weeks. I&#8217;m finding that it takes me quite a bit longer to write stuff with factor than with other languages, but the process is enjoyable and I get the feeling that I&#8217;m learning something useful each time. The question is: will programming speed improve with experience. [..." _xmlchars "I&#8217;ve been playing with Factor for a couple of weeks. I&#8217;m finding that it takes me quite a bit longer to write stuff with factor than with other languages, but the process is enjoyable and I get the feeling that I&#8217;m learning something useful each time. The question is: will programming speed improve with experience. [...]"[255] "More factor: tabular to triples"[1] description "I&#8217;ve been playing with Factor for a couple of weeks. I&#8217;m finding that it takes me quite a bit longer to write stuff with factor than with other languages, but the process is enjoyable and I get the feeling that I&#8217;m learning something useful each time. The question is: will programming speed improve with experience. [..." "

I&#8217;ve been playing with Factor for a couple of weeks. I&#8217;m finding that it takes me quite a bit longer to write stuff with factor than with other languages, but the process is enjoyable and I get the feeling that I&#8217;m learning something useful each time. The question is: will programming speed improve with experience. And more to the point, eventually, will I be able to write code faster in factor than in gambit scheme?

" _xmlchars "

I&#8217;ve been playing with Factor for a couple of weeks. I&#8217;m finding that it takes me quite a bit longer to write stuff with factor than with other languages, but the process is enjoyable and I get the feeling that I&#8217;m learning something useful each time. The question is: will programming speed improve with experience. And more to the point, eventually, will I be able to write code faster in factor than in gambit scheme?

Here&#8217;s a task I&#8217;ve been trying to accomplish the past couple of evenings: converting tabular data into triples. I&#8217;ve written this functionality before in both python and scheme before so it&#8217;s a good exercise for developing factor experience.

First, here&#8217;s the code in python:

def row_to_triples(i,cols,row):    return [ (i,col,cell) for col,cell in zip(cols,row) ]def tabular_to_triples(startid,cols,rows):    triples = []    for i,row in zip(range(startid,startid+len(rows)),rows):        triples += row_to_triples(i, cols, row)    return triples

Which works like this:

>> print tabular_to_triples(0,                [\"col1\",\"col2\",\"col3\"],                [[\"a\",\"b\",\"c\"],[\"e\",\"f\",\"g\"]])[(0, 'col1', 'a'), (0, 'col2', 'b'), (0, 'col3', 'c'), (1, 'col1', 'e'), (1, 'col2', 'f'), (1, 'col3', 'g')]

Here&#8217;s the equivalent in gambit scheme using srfi-42 eager comprehensions:

(include \"srfi-42.scm\")(define (tabular->triples startid rows cols)  (define (row->triples id cols row)    (list-ec (:parallel (:list i cols) (:list j row))             (list id i j)))  (append-ec (:list row (index i) rows)             (row->triples (+ i startid) cols row)))

Which yields:

>> (tabular->triples 0                     '((a b c)(d e f))                     '(col1 col2 col3))((0 col1 a) (0 col2 b) (0 col3 c) (1 col1 d) (1 col2 e) (1 col3 f))

ASIDE: If you&#8217;re evaluating gambit scheme (or scheme in general) be sure to check out srfi-42 and also Alex Shinn&#8217;s ml-style pattern matching module from http://synthcode.com/scheme/ . Gambit scheme out-of-the-box is like abstraction assembly language - you need tools layered over the top for real world use.

Ok, so on to factor. My first attempt was:

: row>triples ( cols row rowid -- triples )  [ -rot 3array ] curry 2map ;: process-row ( rowid cols row -- rowid cols triples )  rot 1 + swapd                           ! inc rownumber  [ swapd row>triples ] 2keep swap rot ;: tabular>triples ( start-rowid cols rows -- triples )  [ process-row ] map concat 2nip ;

Which works like this:

>> { \"c1\" \"c2\" \"c3\" } { { \"a\" \"b\" \"c\" } { \"d\" \"e\" \"f\" } } 0 tabular>triples>> .{    { 1 \"c1\" \"a\" }    { 1 \"c2\" \"b\" }    { 1 \"c3\" \"c\" }    { 2 \"c1\" \"d\" }    { 2 \"c2\" \"e\" }    { 2 \"c3\" \"f\" }}

This is reasonably compact, but looks horrible to me and is pretty difficult to follow due to all the stack shuffling. I&#8217;m starting to learn that the way to eliminate stack shuffling is to use combinators, and try to factor out the common code.

I did a bit of functional de-composition with the hope that creating words for the pieces would yield clearer code. The pieces of functionality needed were:

- creating a triple from 3 elements on the stack. Handled by 3array
- enumerating a variable whilst mapping through a sequence
- holding a variable whilst mapping through a sequence. Handled by &#8216;curry map&#8217;
- mapping two sequences in parallel (columns and rows). Handled by 2map
- concatenating sequences (rows) of triples together: concat

Actually once I&#8217;d worked out the which words I wanted I found that a most of them already existed. In fact all of them except &#8216;enumerating a variable whilst mapping through a sequence&#8217;, so I wrote &#8216;map-with-counter&#8217; to provide this. It prepends enumerating code to the map quotation before calling map, then cleans up the index variable at the end:

: map-with-counter ( start seq quot -- newseq ) [ [ dup 1+ swap ] dip ] swap compose map nip ;

And so few attempts later and I&#8217;ve got:

: row>triples ( rowid cols row -- triples )  [ 3array ] curry** 2map ;: tabular>triples ( start-rowid cols rows -- triples )  [ row>triples ] curry* map-with-counter concat ;

Which I&#8217;m quite pleased with. However in addition to map-with-counter I&#8217;ve had to create a new partial application combinator: curry**, which is general but doesn&#8217;t exist in the standard library. A sure sign that I&#8217;m doing something wrong, or at least differently to other factor developers.
And while I was writing curry** I ended up creating same new stack shufflers:

: rotd ( a b c d -- b c a d )  >r rot r> ;: -rotd ( a b c d -- c a b d ) >r -rot r> ;: dupdd ( a b c -- a a b c ) >r dupd r> ;! partial application of quot based on 3rd item of stack! see curry, curry*: curry** ( param obj obj quot -- obj obj curry )  rotd [ -rotd call ] 2curry ; inline

The underlying theme to these extra words is dealing with the 3rd element in the stack and below. So does that mean that seasoned factor developers tend to switch to some other mechanism when there&#8217;s more than 3 stack variables in play? Sounds like a question for the mailing list&#8230;

"[255] "More factor: tabular to triples"[1] encoded[1] "

I&#8217;ve been playing with Factor for a couple of weeks. I&#8217;m finding that it takes me quite a bit longer to write stuff with factor than with other languages, but the process is enjoyable and I get the feeling that I&#8217;m learning something useful each time. The question is: will programming speed improve with experience. And more to the point, eventually, will I be able to write code faster in factor than in gambit scheme?

" "More factor: tabular to triples"[1] commentRSS[1] http://www.phildawes.net/blog/2007/10/11/more-factor-tabular-to-triples/feed/ "More factor: tabular to triples"[1] tag item "Phil Dawes' Stuff"[1] item "More factor: tabular to triples"[1] "Baby steps with factor: a csvparser"[1] title "Baby steps with factor: a csvparser" "Baby steps with factor: a csvparser"[1] link http://www.phildawes.net/blog/2007/10/02/baby-steps-with-factor-a-csvparser/ "Baby steps with factor: a csvparser"[1] comments http://www.phildawes.net/blog/2007/10/02/baby-steps-with-factor-a-csvparser/#comments "Baby steps with factor: a csvparser"[1] pubDate "Tue, 02 Oct 2007 21:46:05 +0000" "Baby steps with factor: a csvparser"[1] creator[1] "Phil Dawes"[2] "Baby steps with factor: a csvparser"[1] category Programming[1] "Baby steps with factor: a csvparser"[1] category factor[1] http://www.phildawes.net/blog/2007/10/02/baby-steps-with-factor-a-csvparser/[1] isPermaLink false[255] "Baby steps with factor: a csvparser"[1] guid http://www.phildawes.net/blog/2007/10/02/baby-steps-with-factor-a-csvparser/[1] "I got round to writing my first factor module tonight: a csv parser." _xmlchars "I got round to writing my first factor module tonight: a csv parser.Actually there&#8217;s already a csv parser written in factor by Daniel Ehrenberg, but it&#8217;s been removed from the latest factor releases. I found a copy of that code here but I don&#8217;t know for how long it&#8217;ll stay there. Unfortunately I had two [...]"[255] "Baby steps with factor: a csvparser"[1] description "I got round to writing my first factor module tonight: a csv parser." "

I got round to writing my first factor module tonight: a csv parser.

" _xmlchars "

I got round to writing my first factor module tonight: a csv parser.

Actually there&#8217;s already a csv parser written in factor by Daniel Ehrenberg, but it&#8217;s been removed from the latest factor releases. I found a copy of that code here but I don&#8217;t know for how long it&#8217;ll stay there.

Unfortunately I had two problems with this existing module: The first was that it ran pretty slowly (1M of csv took ~5 seconds to parse on my laptop) mainly because my copy of factor wouldn&#8217;t compile the state-parser module that it depends on so it ran un-optimized. The second was that I needed a parser that could parse a row at a time for reading huge csv files in chunks. I took that as an opportunity to write my own.

The code performs ok (~500ms for 1M csv) and parses all the examples on the wikipedia csv page, but I can&#8217;t help feeling that I&#8217;ve written it in a similar style to what I would have done if I were using scheme. If anybody has any hints on ways to make the code smaller or faster or more elegant then I&#8217;d be delighted.

USING: kernel sequences io namespaces combinators ;IN: csvparserDEFER: quoted-field: not-quoted-field ( -- endchar )  \",\\"nst\" read-until   #! \"  dup  { { CHAR: s  [ drop % not-quoted-field ] } ! skip whitespace    { CHAR: t  [ drop % not-quoted-field ] }    { CHAR: ,   [ swap % ] }    { CHAR: \"   [ drop drop quoted-field ] }  ! \"    { CHAR: n  [ swap % ] }    { f         [ swap % ] }       ! eof  } case ;: maybe-escaped-quote ( -- endchar )  read1  dup  { { CHAR: \"   [ , quoted-field ] }     ! \" is an escaped quote    { CHAR: s  [ drop not-quoted-field ] }    { CHAR: t  [ drop not-quoted-field ] }    [ drop ]  } case ;: quoted-field ( -- endchar )  \"\\"\" read-until                                 ! \"  drop % maybe-escaped-quote ;: field ( -- string sep )  [ not-quoted-field ] \"\" make swap ;: (row) ( -- sep )  field swap ,  dup CHAR: , = [ drop (row) ] when ;: row ( -- array[string] eof? )  [ (row) ] { } make swap ;: (csv) ( -- )  row swap , [ (csv) ] when ;: csv-row ( stream -- row )  [ row drop ] with-stream ;: csv ( stream -- rows )  [ [ (csv) ] { } make ] with-stream ;

If anybody&#8217;s interested the module (inc tests and doc) is here.

"[255] "Baby steps with factor: a csvparser"[1] encoded[1] "

I got round to writing my first factor module tonight: a csv parser.

" "Baby steps with factor: a csvparser"[1] commentRSS[1] http://www.phildawes.net/blog/2007/10/02/baby-steps-with-factor-a-csvparser/feed/ "Baby steps with factor: a csvparser"[1] tag item "Phil Dawes' Stuff"[1] item "Baby steps with factor: a csvparser"[1] "And another new programming language"[1] title "And another new programming language" "And another new programming language"[1] link http://www.phildawes.net/blog/2007/09/28/and-another-new-programming-language/ "And another new programming language"[1] comments http://www.phildawes.net/blog/2007/09/28/and-another-new-programming-language/#comments "And another new programming language"[1] pubDate "Fri, 28 Sep 2007 11:38:02 +0000" "And another new programming language"[1] creator[1] "Phil Dawes"[3] "And another new programming language"[1] category workfriendly "And another new programming language"[1] category Programming[2] "And another new programming language"[1] category scheme "And another new programming language"[1] category factor[2] "And another new programming language"[1] category forth http://www.phildawes.net/blog/2007/09/28/and-another-new-programming-language/[1] isPermaLink false[255] "And another new programming language"[1] guid http://www.phildawes.net/blog/2007/09/28/and-another-new-programming-language/[1] "Last year I embarked on somwhat of a journey to find a better language for my home projects after getting a bit frustrated by python&#8217;s lack of blocks and general cruftyness. After a couple of months of trying various different things I settled on Gambit Scheme for my spare-time data indexing project. A minimal core [..." _xmlchars "Last year I embarked on somwhat of a journey to find a better language for my home projects after getting a bit frustrated by python&#8217;s lack of blocks and general cruftyness. After a couple of months of trying various different things I settled on Gambit Scheme for my spare-time data indexing project. A minimal core [...]"[255] "And another new programming language"[1] description "Last year I embarked on somwhat of a journey to find a better language for my home projects after getting a bit frustrated by python&#8217;s lack of blocks and general cruftyness. After a couple of months of trying various different things I settled on Gambit Scheme for my spare-time data indexing project. A minimal core [..." "

Last year I embarked on somwhat of a journey to find a better language for my home projects after getting a bit frustrated by python&#8217;s lack of blocks and general cruftyness. After a couple of months of trying various different things I settled on Gambit Scheme for my spare-time data indexing project. A minimal core language with uniform syntax and macros. Lots of potentual for adapting and building language features that make sense to me. Sorted.

" _xmlchars "

Last year I embarked on somwhat of a journey to find a better language for my home projects after getting a bit frustrated by python&#8217;s lack of blocks and general cruftyness. After a couple of months of trying various different things I settled on Gambit Scheme for my spare-time data indexing project. A minimal core language with uniform syntax and macros. Lots of potentual for adapting and building language features that make sense to me. Sorted.

Then last week I got round to reading Richard Jones&#8217;s minimal forth code. A language compiler and runtime in a couple of pages of well documented X86 assembler; small enough to read and understand the whole system in a bus journey.

Like Scheme, Forth has the ability to construct new language features, allowing user code to get between the parser and the evaluator to modify the language itself. Also, like Scheme, it has a uniform syntax - words separated by spaces, which makes re-writing code on the fly practical. Both these features mean that a fully fledged programming environment can be bootstrapped up from a minimal core. And the hook is: that minimal core is so much smaller for Forth than it is for Scheme.

One thing lead to another and I got interested in stack languages. Now I&#8217;m looking at Factor and wondering&#8230;

Factor has ticks in all the right boxes: minimal core, machine code compiler, macros, continuations, lightweight threads and message passing concurrency. It&#8217;s also the tersest language I&#8217;ve seen - code to do something always seems a fraction of the size I expect it to be. On the other hand it&#8217;s so radically different to anything else I&#8217;ve programmed in and it makes my head hurt. Could it be a contender to knock Scheme off the top spot?

"[255] "And another new programming language"[1] encoded[1] "

Last year I embarked on somwhat of a journey to find a better language for my home projects after getting a bit frustrated by python&#8217;s lack of blocks and general cruftyness. After a couple of months of trying various different things I settled on Gambit Scheme for my spare-time data indexing project. A minimal core language with uniform syntax and macros. Lots of potentual for adapting and building language features that make sense to me. Sorted.

" "And another new programming language"[1] commentRSS[1] http://www.phildawes.net/blog/2007/09/28/and-another-new-programming-language/feed/ "And another new programming language"[1] tag item "Phil Dawes' Stuff"[1] item "And another new programming language"[1] "JQuery Rocks!"[1] title "JQuery Rocks!" "JQuery Rocks!"[1] link http://www.phildawes.net/blog/2007/09/17/jquery-rocks/ "JQuery Rocks!"[1] comments http://www.phildawes.net/blog/2007/09/17/jquery-rocks/#comments "JQuery Rocks!"[1] pubDate "Mon, 17 Sep 2007 14:47:56 +0000" "JQuery Rocks!"[1] creator[1] "Phil Dawes"[4] "JQuery Rocks!"[1] category workfriendly[1] "JQuery Rocks!"[1] category JavaScript "JQuery Rocks!"[1] category ajax "JQuery Rocks!"[1] category browser http://www.phildawes.net/blog/2007/09/17/jquery-rocks/[1] isPermaLink false[255] "JQuery Rocks!"[1] guid http://www.phildawes.net/blog/2007/09/17/jquery-rocks/[1] "JQuery is a javascript library providing a browser API in the functional style. In particular it supplies a CSS style selector language which makes the code for manipulating DOM objects simple and terse." _xmlchars "JQuery is a javascript library providing a browser API in the functional style. In particular it supplies a CSS style selector language which makes the code for manipulating DOM objects simple and terse.Simon Willison does a great job of introducing the library and explaining why it&#8217;s so cool. For a taster, here&#8217;s jquery line which [...]"[255] "JQuery Rocks!"[1] description "JQuery is a javascript library providing a browser API in the functional style. In particular it supplies a CSS style selector language which makes the code for manipulating DOM objects simple and terse." "

JQuery is a javascript library providing a browser API in the functional style. In particular it supplies a CSS style selector language which makes the code for manipulating DOM objects simple and terse.

" _xmlchars "

JQuery is a javascript library providing a browser API in the functional style. In particular it supplies a CSS style selector language which makes the code for manipulating DOM objects simple and terse.

Simon Willison does a great job of introducing the library and explaining why it&#8217;s so cool. For a taster, here&#8217;s jquery line which loads html into a div with id &#8216;intro&#8217;.

$('div#intro').load('/some/fragment.html');

In particular I found that using CSS selectors to locate and manipulate DOM objects coupled with jquery&#8217;s terseness meant it was just as easy to attach events to browser from the javascript code than by adding &#8216;onclick&#8217; handlers in the HTML. This nicely seperates the HTML from the javascript and lets you think about how the page will behave on devices without AJAX and javascript capability.

"[255] "JQuery Rocks!"[1] encoded[1] "

JQuery is a javascript library providing a browser API in the functional style. In particular it supplies a CSS style selector language which makes the code for manipulating DOM objects simple and terse.

" "JQuery Rocks!"[1] commentRSS[1] http://www.phildawes.net/blog/2007/09/17/jquery-rocks/feed/ "JQuery Rocks!"[1] tag item "Phil Dawes' Stuff"[1] item "JQuery Rocks!"[1] "Indexing dilema - in memory or on disk?"[1] title "Indexing dilema - in memory or on disk?" "Indexing dilema - in memory or on disk?"[1] link http://www.phildawes.net/blog/2007/07/30/indexing-dilema-in-memory-or-on-disk/ "Indexing dilema - in memory or on disk?"[1] comments http://www.phildawes.net/blog/2007/07/30/indexing-dilema-in-memory-or-on-disk/#comments "Indexing dilema - in memory or on disk?"[1] pubDate "Mon, 30 Jul 2007 06:16:35 +0000" "Indexing dilema - in memory or on disk?"[1] creator[1] "Phil Dawes"[5] "Indexing dilema - in memory or on disk?"[1] category "Semantic Web" "Indexing dilema - in memory or on disk?"[1] category tagtriples http://www.phildawes.net/blog/2007/07/30/indexing-dilema-in-memory-or-on-disk/[1] isPermaLink false[255] "Indexing dilema - in memory or on disk?"[1] guid http://www.phildawes.net/blog/2007/07/30/indexing-dilema-in-memory-or-on-disk/[1] "I&#8217;m trying to work out if it&#8217;s possible to get the index searching performance I want using a disk-backed store, or whether I need to focus on optimising the indexes to fit totally in memory. The problem is that the optimisation strategies are somewhat different:" _xmlchars "I&#8217;m trying to work out if it&#8217;s possible to get the index searching performance I want using a disk-backed store, or whether I need to focus on optimising the indexes to fit totally in memory. The problem is that the optimisation strategies are somewhat different:Storing indexes on disk:- Increase redundancy, trading space for better locality [...]"[255] "Indexing dilema - in memory or on disk?"[1] description "I&#8217;m trying to work out if it&#8217;s possible to get the index searching performance I want using a disk-backed store, or whether I need to focus on optimising the indexes to fit totally in memory. The problem is that the optimisation strategies are somewhat different:" "

I&#8217;m trying to work out if it&#8217;s possible to get the index searching performance I want using a disk-backed store, or whether I need to focus on optimising the indexes to fit totally in memory. The problem is that the optimisation strategies are somewhat different:

" _xmlchars "

I&#8217;m trying to work out if it&#8217;s possible to get the index searching performance I want using a disk-backed store, or whether I need to focus on optimising the indexes to fit totally in memory. The problem is that the optimisation strategies are somewhat different:

Storing indexes on disk:
- Increase redundancy, trading space for better locality of reference.

Storing indexes in memory:
- Reduce redundancy, trading locality of reference for fitting more data in memory.

Of course reference locality for caching does have an effect in memory as well as disk, but it&#8217;s more a factor of ~10 (i.e. ~10 times slower to fetch a word from main memory than from l1 cache) whereas a disk seek compared to a sequential read is more like a factor of ~100000.

Typical server machines seem to have the order of ~100 times more disk than memory so whatever happens the redundancy increase needs to be less than a factor of 100 or there&#8217;s little point.

Sequential disk reads are ~10 times slower than random access memory reads (cache misses) and ~100 times slower than sequential (L1) cache reads, so even if there are no seeks this is still going to be 10-100 times slower. But the payoff is potentually a lot more indexed data per server node.

&#8212;

N.B. these numbers are *very* approximate orders of magnitude. Assumptions are from the below links, along with my own measurements using hdparm, bonnie++, seeker.c and a some of my own code. Please let me know if I&#8217;m wildly out with any of these!

http://www.linuxinsight.com/how_fast_is_your_disk.html
http://homepage.virgin.net/roy.longbottom/randmem%20results.htm

"[255] "Indexing dilema - in memory or on disk?"[1] encoded[1] "

I&#8217;m trying to work out if it&#8217;s possible to get the index searching performance I want using a disk-backed store, or whether I need to focus on optimising the indexes to fit totally in memory. The problem is that the optimisation strategies are somewhat different:

" "Indexing dilema - in memory or on disk?"[1] commentRSS[1] http://www.phildawes.net/blog/2007/07/30/indexing-dilema-in-memory-or-on-disk/feed/ "Indexing dilema - in memory or on disk?"[1] tag item "Phil Dawes' Stuff"[1] item "Indexing dilema - in memory or on disk?"[1] "Indexes, Hashes " _xmlchars "Indexes, Hashes & Compression"[255] "Indexes, Hashes "[1] title "Indexes, Hashes " "Indexes, Hashes "[1] link http://www.phildawes.net/blog/2007/07/26/indexes-hashes-compression/ "Indexes, Hashes "[1] comments http://www.phildawes.net/blog/2007/07/26/indexes-hashes-compression/#comments "Indexes, Hashes "[1] pubDate "Thu, 26 Jul 2007 20:56:22 +0000" "Indexes, Hashes "[1] creator[1] "Phil Dawes"[6] "Indexes, Hashes "[1] category "Semantic Web"[1] "Indexes, Hashes "[1] category tagtriples[1] "Indexes, Hashes "[1] category data "Indexes, Hashes "[1] category indexing http://www.phildawes.net/blog/2007/07/26/indexes-hashes-compression/[1] isPermaLink false[255] "Indexes, Hashes "[1] guid http://www.phildawes.net/blog/2007/07/26/indexes-hashes-compression/[1] "The new triplestore is coming along. It can do substring text searches (using a suffix array) and has a basic relational query engine. It doesn&#8217;t optimise the query plans yet, but if you enter the queries in a good order (most selective clauses first) then you get good performance." _xmlchars "The new triplestore is coming along. It can do substring text searches (using a suffix array) and has a basic relational query engine. It doesn&#8217;t optimise the query plans yet, but if you enter the queries in a good order (most selective clauses first) then you get good performance.A few things have changed in my [...]"[255] "Indexes, Hashes "[1] description "The new triplestore is coming along. It can do substring text searches (using a suffix array) and has a basic relational query engine. It doesn&#8217;t optimise the query plans yet, but if you enter the queries in a good order (most selective clauses first) then you get good performance." "

The new triplestore is coming along. It can do substring text searches (using a suffix array) and has a basic relational query engine. It doesn&#8217;t optimise the query plans yet, but if you enter the queries in a good order (most selective clauses first) then you get good performance.

" _xmlchars "

The new triplestore is coming along. It can do substring text searches (using a suffix array) and has a basic relational query engine. It doesn&#8217;t optimise the query plans yet, but if you enter the queries in a good order (most selective clauses first) then you get good performance.

A few things have changed in my thinking since my last post about indexing. Although using hashes to represent tokens is really useful when joining datasets from different nodes in a cluster (no coordination overhead), I&#8217;m now thinking that they&#8217;re not such a good idea for when laying out the actual triple indexes in memory (or on disk).

My reasoning is:

In order to get the performance I want (100 row results from relational queries in ~half second) I&#8217;m either going to have to keep the entire set of indexes in memory, or at the very least minimise the disk activity to a small number of sequential reads. Disk seeks are the order of ~10ms so 50 of them and I&#8217;m shot. If I end up aiming for the all-in-memory approach then I want to cram as many triples in to memory as possible. If I do use disk reads then locality of reference will be fundamental to the layout of the indexes.

Either way, I&#8217;m going to need to use compression on the indexes to achieve optimal storage or read efficiency. The problem with hashes is that they introduce a lot of randomness into the mix which reduces the ability to do delta compression (and then run length encoding of the deltas). I suspect that controlling allocation of identifiers could also be very useful in optimising locality of reference. All this is theoretical at the moment as I haven&#8217;t actually implemented any index compression, but I hope to do this soon.

"[255] "Indexes, Hashes "[1] encoded[1] "

The new triplestore is coming along. It can do substring text searches (using a suffix array) and has a basic relational query engine. It doesn&#8217;t optimise the query plans yet, but if you enter the queries in a good order (most selective clauses first) then you get good performance.

" "Indexes, Hashes "[1] commentRSS[1] http://www.phildawes.net/blog/2007/07/26/indexes-hashes-compression/feed/ "Indexes, Hashes "[1] tag item "Phil Dawes' Stuff"[1] item "Indexes, Hashes "[1] "A poor man"[1] title "A poor man" "A poor man"[1] link http://www.phildawes.net/blog/2007/07/25/a-poor-mans-scheme-profiler/ "A poor man"[1] comments http://www.phildawes.net/blog/2007/07/25/a-poor-mans-scheme-profiler/#comments "A poor man"[1] pubDate "Wed, 25 Jul 2007 22:39:19 +0000" "A poor man"[1] creator[1] "Phil Dawes"[7] "A poor man"[1] category General[1] "A poor man"[1] category Programming[3] "A poor man"[1] category scheme[1] "A poor man"[1] category gambit http://www.phildawes.net/blog/2007/07/25/a-poor-mans-scheme-profiler/[1] isPermaLink false[255] "A poor man"[1] guid http://www.phildawes.net/blog/2007/07/25/a-poor-mans-scheme-profiler/[1] "Gambit scheme lacks a profiler that can profile scheme with embedded C code. (There&#8217;s statprof, but unfortunately it doesn&#8217;t profile embeded C). I needed to do this pretty desperately for my triple indexing stuff so I&#8217;ve written a simple macro which takes wall clock timings of functions and accumulates them. " _xmlchars "Gambit scheme lacks a profiler that can profile scheme with embedded C code. (There&#8217;s statprof, but unfortunately it doesn&#8217;t profile embeded C). I needed to do this pretty desperately for my triple indexing stuff so I&#8217;ve written a simple macro which takes wall clock timings of functions and accumulates them. You replace &#8216;define&#8217; with &#8216;define-timed&#8217; [...]"[255] "A poor man"[1] description "Gambit scheme lacks a profiler that can profile scheme with embedded C code. (There&#8217;s statprof, but unfortunately it doesn&#8217;t profile embeded C). I needed to do this pretty desperately for my triple indexing stuff so I&#8217;ve written a simple macro which takes wall clock timings of functions and accumulates them. " "

Gambit scheme lacks a profiler that can profile scheme with embedded C code. (There&#8217;s statprof, but unfortunately it doesn&#8217;t profile embeded C). I needed to do this pretty desperately for my triple indexing stuff so I&#8217;ve written a simple macro which takes wall clock timings of functions and accumulates them.

" _xmlchars "

Gambit scheme lacks a profiler that can profile scheme with embedded C code. (There&#8217;s statprof, but unfortunately it doesn&#8217;t profile embeded C). I needed to do this pretty desperately for my triple indexing stuff so I&#8217;ve written a simple macro which takes wall clock timings of functions and accumulates them.

You replace &#8216;define&#8217; with &#8216;define-timed&#8217; in the functions you want profiled, and then the time spent in each function is accumulated in a global hash table. It&#8217;s not pretty, but it&#8217;s simple.

The macro needs some supporting code (which I keep in a seperate module so that it can be compiled in order to minimise overhead). :

(define *timerhash* (make-table));;; call this before running the code(define (reset-timer) (set! *timerhash* (make-table)));;; adds the time spent in the thunk to an entry in the hashtable(define (accum-time name thunk)  (let* ((timebefore (real-time))         (res (thunk)))    (table-set! *timerhash* name                (let ((current (table-ref *timerhash* name '(0 0))))                  (list (+ (car current) (- (real-time) timebefore))                        (+ (cadr current) 1))))    res));;; call this afterwards to get the times(define (get-times)  (map (lambda (e) (list (first e)                    (* 1000 (second e))                    (third e)))       (sort-list (table->list *timerhash*)                  (lambda (a b) (> (second a)                              (second b))))))

And here&#8217;s the macro. It basically just wraps the function body in a call to &#8216;accum-time&#8217;:

(define-syntax define-timed  (syntax-rules ()    ((define-timed (name . args) body ...)     (define (name . args)       (accum-time 'name (lambda ()                           body ...))))))

Here&#8217;s some example output. The first number is the accumulated milliseconds spent in the function, and the second is the number of times it was called.

((handle-query-clause 2182.3294162750244 8) (do-substring-search-query 1678.9898872375488 1) (run-query 1678.9379119873047 1) (main 1678.929090499878 1) (handle-substr-search 705.5349349975586 2) (convert-ids-to-symbols 400.3608226776123 1) (handle-s_o-clause 192.81506538391113 2) (lookup-o->s 192.52514839172363 2) (lookup-2-lvl-index 192.48294830322266 2) (join 163.8491153717041 8) (hash-join 163.6500358581543 5) (fetch-int32-results 118.37220191955566 655) (project 100.7072925567627 3) (concat 77.42834091186523 11793) (handle-p->so-clause 55.348873138427734 2) (bounds-matching-1intersect 51.54275894165039 2) (lookup-p->2-lvl-index 50.98700523376465 2) (search-32-32-index 42.65856742858887 94) (cells-at-positions 35.561561584472656 11697) (fetch-32-32-results 1.171112060546875 94) (extract-column .2582073211669922 4) (get-join-columns .05888938903808594 5) (text-search-to-structured-query .032901763916015625 1) (columns .0324249267578125 10) (column-position .031232833862304688 8) (rows .011920928955078125 4))

Hmmm.. I really need to get something to html pretty print the scheme code.

"[255] "A poor man"[1] encoded[1] "

Gambit scheme lacks a profiler that can profile scheme with embedded C code. (There&#8217;s statprof, but unfortunately it doesn&#8217;t profile embeded C). I needed to do this pretty desperately for my triple indexing stuff so I&#8217;ve written a simple macro which takes wall clock timings of functions and accumulates them.

" "A poor man"[1] commentRSS[1] http://www.phildawes.net/blog/2007/07/25/a-poor-mans-scheme-profiler/feed/ "A poor man"[1] tag item "Phil Dawes' Stuff"[1] item "A poor man"[1] "Coding when you"[1] title "Coding when you" "Coding when you"[1] link http://www.phildawes.net/blog/2007/07/23/coding-when-youre-tired-and-unmotivated/ "Coding when you"[1] comments http://www.phildawes.net/blog/2007/07/23/coding-when-youre-tired-and-unmotivated/#comments "Coding when you"[1] pubDate "Mon, 23 Jul 2007 05:29:09 +0000" "Coding when you"[1] creator[1] "Phil Dawes"[8] "Coding when you"[1] category General[2] "Coding when you"[1] category Programming[4] http://www.phildawes.net/blog/2007/07/23/coding-when-youre-tired-and-unmotivated/[1] isPermaLink false[255] "Coding when you"[1] guid http://www.phildawes.net/blog/2007/07/23/coding-when-youre-tired-and-unmotivated/[1] "I&#8217;ve been struggling to motivate myself recently. I spent a lot of my spare time in my twenties programming free stuff after work with no trouble at all, but now in my thirties the energy and enthusiasm seems a little harder to come by. Plus Claire and I are expecting a baby any day now [..." _xmlchars "I&#8217;ve been struggling to motivate myself recently. I spent a lot of my spare time in my twenties programming free stuff after work with no trouble at all, but now in my thirties the energy and enthusiasm seems a little harder to come by. Plus Claire and I are expecting a baby any day now [...]"[255] "Coding when you"[1] description "I&#8217;ve been struggling to motivate myself recently. I spent a lot of my spare time in my twenties programming free stuff after work with no trouble at all, but now in my thirties the energy and enthusiasm seems a little harder to come by. Plus Claire and I are expecting a baby any day now [..." "

I&#8217;ve been struggling to motivate myself recently. I spent a lot of my spare time in my twenties programming free stuff after work with no trouble at all, but now in my thirties the energy and enthusiasm seems a little harder to come by. Plus Claire and I are expecting a baby any day now and the last few months have played havoc with sleeping patterns (nothing like what&#8217;s to come I&#8217;m sure!).

" _xmlchars "

I&#8217;ve been struggling to motivate myself recently. I spent a lot of my spare time in my twenties programming free stuff after work with no trouble at all, but now in my thirties the energy and enthusiasm seems a little harder to come by. Plus Claire and I are expecting a baby any day now and the last few months have played havoc with sleeping patterns (nothing like what&#8217;s to come I&#8217;m sure!).

Still, it&#8217;s definitely possible to crank code on your own projects when you&#8217;re knackered, you just have to be a bit more structured about how you generate motivation and enthusiasm from a body that just wants to watch telly. Note that we haven&#8217;t had the baby yet (any day now!) so all of this may be completely moot once we hit the real deal, but I figured it might be useful to somebody and I haven&#8217;t blogged for a while so&#8230;

The key to programming motivation for me seems to be all about engineering a low barrier to getting started. I really enjoy spare-time software development, but if I don&#8217;t get up to speed quickly then it&#8217;s easy to drift off to doing other things like reading reddit or going through my rss feeds. Here&#8217;s some things I do to keep my eyes on the prize:

Keep a project diary of ideas and thoughts.

I usually have a text file called &#8216;IDEAS&#8217; in the base directory of my projects. In it I keep a diary of thoughts and ideas dated in reverse chronological order (like a blog). Sometimes I noodle in it - write questions I want answered, jot while I think through problems. It is not intended to be read by anybody except me so things are often un-finished, written in shorthand and I rarely write explanations.

The diary serves a number of important purposes in addition to being a low motivation way to get going:

Do &#8216;test driven&#8217; development.

Test driven development is a great technique generally, but I find it especially useful when I&#8217;m too tired to be motivated to program. That also applies when the problem is too big for an obviously good solution to fit in my head and I&#8217;m exhausted just thinking about it.

When say &#8216;test driven&#8217;, I mean really *driven*. The implementation code written really is just enough to pass the tests. In the early stages that means hard-coded results..:

The point is that by the time you end up evolving the meat of the functionality, you&#8217;ve already crystalised a lot of what the implementation needs to do and can tackle it in very small baby steps. Small chunks are good for me when I&#8217;m tired because I don&#8217;t need to hold too much in my head at once. The tests passing provides continual positive feedback which is a good motivator and often that gets me out of a programming rut.

Write poorly performing code and optimize later

It&#8217;s very easy for me to fall into the trap of optimising code early, especially because I&#8217;m keeping a project diary and spending considerably more time thinking about the project than actually implementing it. Unfortunately although it&#8217;s fun to come up with strategies to optimise for performance, coding that way tends to make the implementation code stagnant very quickly and that makes the whole experience very demotivating. To combat this I try to get into the habit of defering performance optimsations by noting them down in the diary instead, and then just concentrate on optimizing for simplicity and code size.

N.B. It&#8217;s worth bearing in mind that profiling and optimizing your code later is really good fun - I have no problem being enthusiastic about making existing code go faster, especially if I have a good set of profiling data to get my teeth into. Also I find that it&#8217;s much more time efficient to build something simply and then profile and optimize later. That&#8217;s really important if you&#8217;re doing this on scraps of spare time. I can&#8217;t emphasise this enough: much more time efficient.

Any more?

So that&#8217;s it. I&#8217;d be really interested to hear of any other spare-time-programming motivation tips that people use. With the baby coming I&#8217;m going to need all the help I can get!

"[255] "Coding when you"[1] encoded[1] "

I&#8217;ve been struggling to motivate myself recently. I spent a lot of my spare time in my twenties programming free stuff after work with no trouble at all, but now in my thirties the energy and enthusiasm seems a little harder to come by. Plus Claire and I are expecting a baby any day now and the last few months have played havoc with sleeping patterns (nothing like what&#8217;s to come I&#8217;m sure!).

" "Coding when you"[1] commentRSS[1] http://www.phildawes.net/blog/2007/07/23/coding-when-youre-tired-and-unmotivated/feed/ "Coding when you"[1] tag item "Phil Dawes' Stuff"[1] item "Coding when you"[1] "A simple scheme unittest DSL"[1] title "A simple scheme unittest DSL" "A simple scheme unittest DSL"[1] link http://www.phildawes.net/blog/2007/05/23/a-simple-scheme-unittest-dsl/ "A simple scheme unittest DSL"[1] comments http://www.phildawes.net/blog/2007/05/23/a-simple-scheme-unittest-dsl/#comments "A simple scheme unittest DSL"[1] pubDate "Wed, 23 May 2007 16:49:29 +0000" "A simple scheme unittest DSL"[1] creator[1] "Phil Dawes"[9] "A simple scheme unittest DSL"[1] category scheme[2] "A simple scheme unittest DSL"[1] category gambit[1] http://www.phildawes.net/blog/2007/05/23/a-simple-scheme-unittest-dsl/[1] isPermaLink false[255] "A simple scheme unittest DSL"[1] guid http://www.phildawes.net/blog/2007/05/23/a-simple-scheme-unittest-dsl/[1] "One of the first things I wrote when I was in the &#8216;nesting&#8217;* phase of learning gambit scheme was a unittest DSL. Part of this was that I wanted an excuse to use r5rs syntax-rules macros, but the real motivation was that I&#8217;d been seduced by the idea of using tests for documentation ala Nat [..." _xmlchars "One of the first things I wrote when I was in the &#8216;nesting&#8217;* phase of learning gambit scheme was a unittest DSL. Part of this was that I wanted an excuse to use r5rs syntax-rules macros, but the real motivation was that I&#8217;d been seduced by the idea of using tests for documentation ala Nat [...]"[255] "A simple scheme unittest DSL"[1] description "One of the first things I wrote when I was in the &#8216;nesting&#8217;* phase of learning gambit scheme was a unittest DSL. Part of this was that I wanted an excuse to use r5rs syntax-rules macros, but the real motivation was that I&#8217;d been seduced by the idea of using tests for documentation ala Nat [..." "

One of the first things I wrote when I was in the &#8216;nesting&#8217;* phase of learning gambit scheme was a unittest DSL. Part of this was that I wanted an excuse to use r5rs syntax-rules macros, but the real motivation was that I&#8217;d been seduced by the idea of using tests for documentation ala Nat Pryce&#8217;s &#8216;Protest&#8217;. Here&#8217;s what I came up with: (example):

" _xmlchars "

One of the first things I wrote when I was in the &#8216;nesting&#8217;* phase of learning gambit scheme was a unittest DSL. Part of this was that I wanted an excuse to use r5rs syntax-rules macros, but the real motivation was that I&#8217;d been seduced by the idea of using tests for documentation ala Nat Pryce&#8217;s &#8216;Protest&#8217;. Here&#8217;s what I came up with: (example):

(define-tests bus-tests  (drive-to-next-stop        ; name of fn/class/symbol being tested    (\"takes bus to next stop\"      (drive-to-next-stop bus)      (assert-equal 'next-stop (bus 'position)))    (\"doesn't stop off at chipshop on the way\"      ; test code to detect chipshop hasn't been stopped at      ))  (pick-up-passengers    (\"picks up passengers from the bus stop\"      test code )    (\"doesn't leave passengers at the stop\"      test code )    (\"waits for old lady running to the stop\"      test code )))

The point is that it&#8217;s easy to write some lisp to traverse this code and generate documentation from it.

I found when using protest in python that the documentation angle reinforced some healthy habits: When you write tests you naturally think &#8216;how would this look to another person?&#8217; &#8216;how can I document the behaviour of this?&#8217; which encourages more complete testing. Also when you look at generated documentation it&#8217;s easy to see which bits you aren&#8217;t testing because the documentation is missing (which then encourages you to write more tests).

The implementation is a bit clunky and makes use of gambit exceptions as a way of terminating tests early because of assert failures (which is a bit rubbish). What probably should be happening is that the outer macro should be re-writing each assert as an &#8216;if&#8217; or something to conditionally execute the rest of the test. (which would portable to other scheme implementations)
To be honest I knocked this up as fast as I could so that I could move onto writing other things (I&#8217;m developing a data aggregation and indexing tool), but the point of this post is more to convey the idea than the implementation.

That said, the crufty (currently gambit specific) implementation is here - hope this is interesting/useful to somebody.

* nesting as in &#8216;building a nest&#8217;

"[255] "A simple scheme unittest DSL"[1] encoded[1] "

One of the first things I wrote when I was in the &#8216;nesting&#8217;* phase of learning gambit scheme was a unittest DSL. Part of this was that I wanted an excuse to use r5rs syntax-rules macros, but the real motivation was that I&#8217;d been seduced by the idea of using tests for documentation ala Nat Pryce&#8217;s &#8216;Protest&#8217;. Here&#8217;s what I came up with: (example):

" "A simple scheme unittest DSL"[1] commentRSS[1] http://www.phildawes.net/blog/2007/05/23/a-simple-scheme-unittest-dsl/feed/ "A simple scheme unittest DSL"[1] tag item "Phil Dawes' Stuff"[1] item "A simple scheme unittest DSL"[1] "Phil Dawes' Stuff"[1] tag channel _:xmlnode-1084-1 channel "Phil Dawes' Stuff"[1] 2.0 version 2.0[255] _:xmlnode-1084-root rss 2.0 "

One of the first things I wrote when I was in the &#8216;nesting&#8217;* phase of learning gambit scheme was a unittest DSL. Part of this was that I wanted an excuse to use r5rs syntax-rules macros, but the real motivation was that I&#8217;d been seduced by the idea of using tests for documentation ala Nat Pryce&#8217;s &#8216;Protest&#8217;. Here&#8217;s what I came up with: (example):

" tag _:xmlnode-1084-138 "One of the first things I wrote when I was in the &#8216;nesting&#8217;* phase of learning gambit scheme was a unittest DSL. Part of this was that I wanted an excuse to use r5rs syntax-rules macros, but the real motivation was that I&#8217;d been seduced by the idea of using tests for documentation ala Nat [..." tag _:xmlnode-1084-137 encoded[1] tag http://purl.org/rss/1.0/modules/content/encoded "Coding when you" tag _:xmlnode-1084-117 "Coding when you"[1] tag _:xmlnode-1084-116 "Last year I embarked on somwhat of a journey to find a better language for my home projects after getting a bit frustrated by python&#8217;s lack of blocks and general cruftyness. After a couple of months of trying various different things I settled on Gambit Scheme for my spare-time data indexing project. A minimal core [..." tag _:xmlnode-1084-59 "

Gambit scheme lacks a profiler that can profile scheme with embedded C code. (There&#8217;s statprof, but unfortunately it doesn&#8217;t profile embeded C). I needed to do this pretty desperately for my triple indexing stuff so I&#8217;ve written a simple macro which takes wall clock timings of functions and accumulates them.

" tag _:xmlnode-1084-114 "JQuery is a javascript library providing a browser API in the functional style. In particular it supplies a CSS style selector language which makes the code for manipulating DOM objects simple and terse." tag _:xmlnode-1084-73 "The new triplestore is coming along. It can do substring text searches (using a suffix array) and has a basic relational query engine. It doesn&#8217;t optimise the query plans yet, but if you enter the queries in a good order (most selective clauses first) then you get good performance." tag _:xmlnode-1084-99 "Indexing dilema - in memory or on disk?"[1] tag _:xmlnode-1084-76 "I got round to writing my first factor module tonight: a csv parser." tag _:xmlnode-1084-44 "

I got round to writing my first factor module tonight: a csv parser.

" tag _:xmlnode-1084-45 "

A nice feature of distributed version control is that you can commit into your repository more often because you aren&#8217;t impacting others with each commit. Recently I&#8217;ve been taking this to the extreme using git and performing a commit almost at every save. I have an emacs key wired for the job: it save the buffers and then runs &#8216;git commit -a -m &#8220;checkpoint&#8221;&#8216;. During a coding session I hammer it frequently.

" tag _:xmlnode-1084-20 "And another new programming language"[1] tag _:xmlnode-1084-47 "

JQuery is a javascript library providing a browser API in the functional style. In particular it supplies a CSS style selector language which makes the code for manipulating DOM objects simple and terse.

" tag _:xmlnode-1084-74 "A simple scheme unittest DSL"[1] tag _:xmlnode-1084-128 commentRSS[1] tag http://wellformedweb.org/CommentAPI/commentRSS "

I&#8217;ve been struggling to motivate myself recently. I spent a lot of my spare time in my twenties programming free stuff after work with no trouble at all, but now in my thirties the energy and enthusiasm seems a little harder to come by. Plus Claire and I are expecting a baby any day now and the last few months have played havoc with sleeping patterns (nothing like what&#8217;s to come I&#8217;m sure!).

" tag _:xmlnode-1084-126 "

The new triplestore is coming along. It can do substring text searches (using a suffix array) and has a basic relational query engine. It doesn&#8217;t optimise the query plans yet, but if you enter the queries in a good order (most selective clauses first) then you get good performance.

" tag _:xmlnode-1084-100 "A poor man"[1] tag _:xmlnode-1084-102 "A poor man" tag _:xmlnode-1084-103 "I&#8217;m trying to work out if it&#8217;s possible to get the index searching performance I want using a disk-backed store, or whether I need to focus on optimising the indexes to fit totally in memory. The problem is that the optimisation strategies are somewhat different:" tag _:xmlnode-1084-85 "

I&#8217;m trying to work out if it&#8217;s possible to get the index searching performance I want using a disk-backed store, or whether I need to focus on optimising the indexes to fit totally in memory. The problem is that the optimisation strategies are somewhat different:

" tag _:xmlnode-1084-86 "JQuery Rocks!"[1] tag _:xmlnode-1084-62 "More factor: tabular to triples"[1] tag _:xmlnode-1084-22 "

Last year I embarked on somwhat of a journey to find a better language for my home projects after getting a bit frustrated by python&#8217;s lack of blocks and general cruftyness. After a couple of months of trying various different things I settled on Gambit Scheme for my spare-time data indexing project. A minimal core language with uniform syntax and macros. Lots of potentual for adapting and building language features that make sense to me. Sorted.

" tag _:xmlnode-1084-60 "I&#8217;ve been struggling to motivate myself recently. I spent a lot of my spare time in my twenties programming free stuff after work with no trouble at all, but now in my thirties the energy and enthusiasm seems a little harder to come by. Plus Claire and I are expecting a baby any day now [..." tag _:xmlnode-1084-125 "Indexes, Hashes "[1] tag _:xmlnode-1084-88 "Indexes, Hashes " tag _:xmlnode-1084-89 "Baby steps with factor: a csvparser"[1] tag _:xmlnode-1084-35 "Gambit scheme lacks a profiler that can profile scheme with embedded C code. (There&#8217;s statprof, but unfortunately it doesn&#8217;t profile embeded C). I needed to do this pretty desperately for my triple indexing stuff so I&#8217;ve written a simple macro which takes wall clock timings of functions and accumulates them. " tag _:xmlnode-1084-113 "A nice feature of distributed version control is that you can commit into your repository more often because you aren&#8217;t impacting others with each commit. Recently I&#8217;ve been taking this to the extreme using git and performing a commit almost at every save. I have an emacs key wired for the job: it save the [..." tag _:xmlnode-1084-19 "

I&#8217;ve been playing with Factor for a couple of weeks. I&#8217;m finding that it takes me quite a bit longer to write stuff with factor than with other languages, but the process is enjoyable and I get the feeling that I&#8217;m learning something useful each time. The question is: will programming speed improve with experience. And more to the point, eventually, will I be able to write code faster in factor than in gambit scheme?

" tag _:xmlnode-1084-33 creator[1] tag http://purl.org/dc/elements/1.1/creator "I&#8217;ve been playing with Factor for a couple of weeks. I&#8217;m finding that it takes me quite a bit longer to write stuff with factor than with other languages, but the process is enjoyable and I get the feeling that I&#8217;m learning something useful each time. The question is: will programming speed improve with experience. [..." tag _:xmlnode-1084-32 "Frequent code checkpointing with git"[1] tag _:xmlnode-1084-9 "Phil Dawes' Stuff"[1] tag _:xmlnode-1084-2 2.0 tag _:xmlnode-1084-1