Saturday, December 31, 2011

Amit Rathore's Introduction to Clojure: Week 3


Shit, I've fallen behind, but thank goodness I'm still learning. Class is moving along a little too quickly for me and I'm not finding enough explanation in Amit's Clojure in Action book's first half.

I've realized that I needed to take a step back and fill in some of the gaps in my mind. I did this by reading Apress' Practical Clojure by Luke VanderHart and Stuart Sierra, which I'll refer to as the PC book in this post. I still need to dig into the state, parallel programming, macro, protocols and performance chapters, but it really helped explain some things I just wasn't understanding from Amit's book.

It will be interesting to see what the second half of the Clojure in Action is about, since that is where it seems to differentiates itself from the other beginner manuals. So, instead of talking about Java interop and concurrency this week, which is what we covered in class, I'll instead go over some of the materials I highlighted while reading the PC book, which further illuminated points I hadn't fully understood.

Tail Recursion

I didn't realize the recur form in clojure was what is used to explicitly use tail call optimization so you don't blow the stack. In Scheme this happens automatically whenever a recursive call is in tail position, not so with Clojure, but then you know that you are in tail recursion when you use recur!

And here is is a great explanation of why tail call recursion is able to optimized, from page 39 in PC, "If the return value of the “outer” function is wholly delegated to the “inner” function, the call is in tail position. If the “outer” function does anything with the value returned from the inner function except just return it, it is not tail recursive and cannot be optimized. This makes sense when the nature of the call stack is considered; if a call is in tail position, then the program can effectively “forget” that it was called recursively at all and delegate the entire program flow to the result of the inner function. If there is additional processing to do, the compiler can’t throw away the outer function. It has to keep it around in order to finish computing its result."

I also benefited from seeing this refactor using the loop special form to show how it is much more compact.

Here is Newton's algorithm to recursively calculate the square root of a number:

(defn abs
   "Calculates the absolute value of a number"
   [n]
   (if (< n 0)
      (* -1 n)
      n))

(defn avg
   "returns the average of two arguments"
   [a b]
   (/ (+ a b) 2))

(defn good-enough?
   "Tests if a guess is close enough to the real square root"
   [number guess]
   (let [diff (- (* guess guess) number)]
      (if (< (abs diff) 0.001)
         true
         false)))

(defn sqrt
   "returns the square root of the supplied number"
   ([number] (sqrt number 1.0))
   ([number guess]
   (if (good-enough? number guess)
      guess
      (sqrt number (avg guess (/ number guess))))))

Now check out the refactor of the sqrt function using loop making the code a little tighter:

(defn loop-sqrt
   "returns the square root of the supplied number"
   [number]
   (loop [guess 1.0]
      (if (good-enough? number guess)
         guess
         (recur (avg guess (/ number guess))))))


Closures 

On page 45 in PC an example is given of a first-class function called rangechecker. The function is first-class because it returns a function, and you can think of the rangechecker function as a "function factory":

(defn rangechecker
   "Returns a function that determines if a number is in a provided range."
   [min max]
   (fn [num]
      (and (<= num max)
           (<= min num))))

To use the rangechecker function you can assign the function to a symbol in your REPL, then call it with a it's single arity parameter, which needs to be a number:

user=> (def myrange (rangechecker 2 10))
#'user/myrange

user=> (myrange 5)
true

The definition for a closure given on page 46 of PC is that they are first-class functions that contain values as well as code. The rangechecker function is an example of this.

Currying

Page 46 of PC also talks about currying and the use of the partial function to illustrate the concept. Currying refers to the process of transforming a function into a function with fewer arguments by wrapping it in a closure. Since that is pretty heavy for me to understand just by reading I was thankful for this example using partial, which incidentally also helped me better understand how to use partial:

user=> (def times-pi (partial * 3.14159))
#’user/times-pi

user=> (times-pi 2)
6.28318

This could have been written by manually definining the following function, which is really just a wrapper for the multiplication function, supplying specific values. Sounds like a closure, doesn't it:

(defn times-pi
   “Multiplies a number by PI”
   [n]
   (* 3.14159 n))

So now partial makes sense to me and I have a better understanding of what closure are used for and what it means when someone talks about currying and that it has nothing to do with Indian food.

Sequences 

I'm glad the PC book had it's own chapter on sequences, because I had not totally understood how they were used, why and what exactly they were. From page 73, "Sequences are simply a way to read, write and modify a data structure that is logically a collection of items."

Not only was this a great chapter describing sequences in detail it was followed by a chapter on the sequence API, which further explained the benefits one could get out of using sequences. I didn't realize a lot of the function we were using in our exercises such as range, partition, map, first, second, last, rest, reduce, apply are all part of the sequence API. Ah, my synapses were firing in sequence after reading these two chapters from the PC book.

In looking at the anatomy of a sequence I was able to finally make the bridge between Lisp's car and cdr functions, which are the equivalents to Clojure's first and rest. They were just renamed to make more sense for modern programmers. The sequence golden rule is that conceptually all sequences share the conceptual model of being a singly-linked list, implemented in terms of first and rest.

To construct a sequence you can use the cons function, which stands for construct. A sequence created by cons is known as a "cons cell", which is just a simple first/rest pair. So here is how we chain multiple cons cells together programmatically, which just so happens to be a common Lisp idiom:

(defn make-int-seq [max]
   (loop [acc nil n max]
      (if (zero? n)
         acc
         (recur (cons n acc) (dec n)))))

user=> (make-int-seq 5)
(1 2 3 4 5)

The sequence chapter helped me get a firm understanding of laziness. "Briefly stated lazy sequences are a highly efficient way to operate on data too large to fit in system memory at once." (Page 77, PC)

This handy little combination of print statements and using the map function from the sequence API, which like most of the higher order functions in the API returns a lazy sequence is highly insightful into the behavior of a lazy sequence.

First a simple squaring function is defined, which produces print side-effects:

(defn square [x]
   (do
      (println (str "Processing: " x))
      (* x x)))

To look at the behavior in your REPL try the following.

Assign the result of a call to the map function using square to a symbol. Then use nth to get a specific item in the list. Call nth again and you will see that the result was cached. Now print the map-result symbol and you will see only values that were not yet cached are realized.

user =>(def map-result (map square '(1 2 3 4 5))
#'user/map-result

user => (nth map-result 2)

user => (nth map-result 2)

user => (println map-result)


When using a lazy sequence calls are not made until they are absolutely required, the point at which the system needs to realize the lazy values.

Another great example which illustrates how lazy sequences can eat memory if there are reference to the realized sequence is given on page 82 in PC.

This version eats up heap space.
When nth is called, the sequence is realized and references are created:

user=> (def integers (iterate inc 0))
#'user/integers

user=> (nth integers 1000000)
1000000


Here is the more efficient version using nth, which does not maintain references, by skipping the binding to the integers symbol:

user=> (nth (iterate inc 0) 1000000)
1000000

State and Parallel Programming

I still have more to learn regarding state and concurrency so I have little to say on the matter at this point in time. However the following explanation of the coordinated state of refs, made a lot of sense to me:

"The most common example of coordinated state is a transfer of funds between two bank accounts: money deposited into one account must also be subtracted from the others, and these two action must occur as a single, coordinated event, or not at all." (Page 97, PC)

Yay, for examples! I also really appreciate the follow-up chapter on state in Chapter 11, Parallel Programming of the PC book, which talks about how to perform concurrent operations, since the discussion of state alone does not address how a program becomes parallel and the strategies involved with doing this.


So that is where I'm at so far in week three of Amit's class. I'm feeling better about my understanding, but this detour to better understand the fundamentals has left me behind. It's now time to catch up and work on the assignments, which I've found really helps to imprint the new concepts. I can only learn so much from reading, at some point I have to switch over to doing, which is a whole different way of using the brain. Time to code!!!

Wednesday, December 21, 2011

Amit Rathore's Introduction to Clojure: Week 2


This week we learned about Clojure's multimethods, which is Chapter 4 in Amit's book Clojure in Action. In this chapter we learn about polymorphism, the use of single dispatch in other languages, and the concept of multiple dispatch and then Clojure's multimethods.

So after reading the chapter I felt I needed to get really grounded with polymorphism. The book refers to the Cardelli/Wegner paper, linked below, but I also found this article by Dan Umbarger as a great way to get my mind in gear for thinking about types and polymorphism which connected new synapses readying me to learn more about the concept of dispatching.

Then I decided to look into the visitor pattern some more and see if I could get a better understanding of why you might use dispatch in a program. I found these papers and articles contained helpful information.

After reading all this I appreciate the simplicity with which Clojure's multimethods handle multiple dispatch, but I'll admit, I'm not sure when to make the design decision to implement them, so I've still got a little more learning to do.

This week Amit had us implement a fantasy banking system with some crazy specs and his example code is posted on Pastie.org. When I have time I'll come back to this post to dissect his code and get a better understanding of the code.

While I was able to implement my own version of a half-solution this week, I still did not complete the assignment and had a lot of difficulties. It definitely was easier than the first assignment though as I'm getting more familiar with Clojure. Here is what I've come up with thus far:

(def coin [:heads :tails])
(def acct-type [:checking :savings :moneymarket])
(def overdraft-acct {:overdraft {:balance 100000 :type :overdraft}})

(defn randomize [objects]
    (rand-nth objects))

(defn coin-flip []
    (randomize coin))

(defn rand-transaction []
    (rand-nth [:withdrawal :deposit]))

(defn create-accounts [n]
  (reduce (fn [mymap mykey] (assoc mymap mykey {:balance 500 :type (randomize acct-type) :transactions 0})) 
                      {} 
                      (for [x (range 1 (+ n 1))] (str "account" x) )))

(defn get-transaction []
  (randomize (filter #(zero? (mod % 10))
     (take (- 400 1) (iterate inc 100)))))

(defn get-account [n] 
  (randomize (keys (create-accounts n))))

(defn interest-amount [percentage acct]
  (float (* 0.01 percentage (acct :balance))))

(defmulti apply-interest :type)

(defmethod apply-interest :checking [acct]
  (interest-amount 0.02 acct))
(defmethod apply-interest :savings [acct]
  (interest-amount 0.04 acct))
(defmethod apply-interest :moneymarket [acct]
  (interest-amount 0.06 acct))

(defn update-trans [accts acct]
  (update-in accts [acct :transactions] + 1))

(defn check-interest-trans [accts acct]
  (let [myaccts (update-trans accts acct)]
  (if (= 0 (mod (get-in accts [acct :transactions]) 2))
      (update-in myaccts [acct :balance] * (apply-interest (myaccts acct)))
     myaccts)))

(defn process-trans [accts n]
  (let [acct (get-account n)
        myaccts (check-interest-trans accts acct)
        trans-type (rand-transaction)]
    (cond 
      (= :withdrawal trans-type)(update-in myaccts [acct :balance] - (get-transaction))
      (= :deposit trans-type) (update-in myaccts [acct :balance] + (get-transaction))
      :default (println "no match"))
      ))

(defn run-fantasy [n]
  (let [acct-count n]
             (loop [accts (create-accounts n)
                    transactions (* n 100)]
                     (if (= 0 transactions)
                       (println "\nloop done")
                       (do
                         (dotimes [x n]
                           (if (> (get-in accts [(str "account" (+ x 1)) :transactions]) 0)
                           (print
                             (str "\n trans" transactions ", acct" (+ x 1)": ")
                             (str "bal: ")(get-in accts [(str "account" (+ x 1)) :balance])
                             (str "tot-trans: ")(get-in accts [(str "account" (+ x 1)) :transactions]))))
                         (recur (process-trans accts acct-count) (dec transactions)))))))
               
(run-fantasy 10)

Stay tuned for next week: Jave interop and Concurrency, Chapters 5 & 6...

Monday, December 19, 2011

My Personal Information Management System

The reason for writing this post is to share my current implementation of a system I use to simplify my digital life wishing to help others and hoping to offer a shortcut to the process of getting things done. The system may seem complicated, and indeed took me years to refine, but the methods are now second nature for me. I continue to evaluate the system and look for improvements and am curious what others are employing successfully in order to manage their daily dose of information.

Let me start with a bit of personal history before I go into the system itself. My first digital personal information management (PIM) tool was a Palm III, which I bought sometime in 1998 for around $399. The Palm platform served my very well for many years. To be able to have your contacts synced to your mobile, make calls, send emails, that was the future, and is today's reality, with prolific smartphone markets and many free network services.

The Palm brought organization and clarity to calendaring, contacts, tasks and notes that previously were in the day planner. Those four areas are the cornerstone of the system I use, along with the concepts that were outlined in David Allen's Getting Things Done (GTD) book.

My first accomplishment using Allen's GTD concepts was to bring my email inbox down to zero messages. I have always kept my personal and work inboxes separate and it took me about a year to get down to the zero mark for both. In the beginning of my GTD practices the inbox would occasionally jump back up to several hundred messages. These setbacks would set things off track for a little while, but I stuck to the method of taking time to get things under control and  always managed to make  my way back to zero after the first time.


My Gmail Inbox

The discipline of getting my inbox to zero required practice, but this was the training I needed to begin seriously gaining control over the information flood. If I could manage data flow in one area, I knew I could repeat the process, thereby freeing up more of my time, which the true gold.

I've now spent over  a decade finding ways which I think increase my productivity, simplify my life, and help me stay more in balance. Simplifying things in your digital life has a high reward potential if you are willing to put in the work upfront.

After years of work keeping my inbox as close to zero as possible, the GTD practice has become rather effortless. Rarely do I have more than 25 emails in my work or personal inbox, and like a well-trained muscle, the process of purging is not painful, particularly since there is nothing "out of control". I now enjoy being able to keep up on not getting behind and flexing new muscles on other problems.

In this post I will share the tools I am using today and how I leverage them to make GTD concepts work for me. This flowchart from Francis Heylighen and Clément Vidal's, "Getting Things Done: The Science behind Stress-Free Productivity" research paper, elegantly illustrates the information capturing process.

Francis Heylighen and Clément Vidal's GTD Flowchart

Here are the tools which I use to catch the many inputs in my life and get the streams organized as quickly as possible without becoming overwhelmed with data.

The Google Trifecta: Gmail/contacts, Gcal, Gdocs 
Google's key concept for organization are views. Without sensible views into your data you have no organization. Views are similar to folders or containers, but the beauty of a view is that an object can live in more than one place at a time and that the view is virtual. For me the key to organizing my email is to have a simple, fairly flat, folder structure, which I can easily sort mail into, without thinking about very much.

As you'll come to see my method for organizing data involves the hierarchical organization of the folders, views and groupings I'm using, which ease archival and retrieval of data. The structures have important differences to me, based on their application and context. By seeing my hierarchies perhaps you might be able to shortcut what has taken me multiple refactors and many years to perfect.

Email
My high level email folder structure looks like this:
  • acquaintance
  • biz
  • diy
  • fam
  • friends
  • personal
  • spirit
  • sys
As a rule of thumb each contact or business I correspond with gets their own folder. I generally only have one level of sub folders in my email. This gives me all the flexibility I need to easily archive messages and find them when I need them later on.

I use Gmail keystroke features, it is easy to jump to folders using the "g+l" quick keys. In case you don't use keystrokes, I recommend turning them on where possible and learning at a minimum how to bring up the help menu, so you can begin to replace your most used clicks with keystrokes. (add this as a task if you need to)

Calendaring
I use the following Google Calendar sharing setup to help me organize and separate the views for appointments and events in my life. When I talk about permissions, I am referring to the permission my spouse and I find work well for us. You likely have similar people in your life for which you can create a shared calendaring model to ease the burden of tracking time and appointments.
  • Main Calendar (read-only to spouse)
  • Spouse's shared main calendar (read-only to me)
  • Joint calendar with my spouse (both can edit)
  • Well-being
  • Maintenance (both can edit)
  • Toodledo iCal (manually added iCal feed from Toodledo)
  • Contacts' birthdays (added from Google special calendars)
  • Holidays (added from Google special calendars)
My main calendar, syncs with my work outlook calendar using Google Calendar Sync. The well-being calendar records activities I like to track like exercise, meditation, journaling and things that are good for my well-being. The maintenance calendar is a place to put things that re-occur around the house, subscriptions, and yearly reminders.Given certain life circumstances I have some calendars come and go based on my tracking needs. It is great to be able to turn on and off any of the calendars, to see just the amount of information I need to see.

My Calendar(s)

Contacts
Here are my high level correspondence contact groups. If you have two monitors you may find it handy to open another Gmail window and switch it to Contacts, and have this on your second monitor if you are working on updating contacts and need to also use your email window.
  • Acquaintance
  • Automated
  • Business
  • Family
  • Friends
  • Medical
  • Pastlife
  • Groups
Three of the above groups are further broken out, using the following naming conventions:
  • Friends + "Context" (e.g. FriendsWork, FriendsSoccer)
  • Family + "Location/Familyname"
  • Groups + "Groupname"
I use the "automated" category for account linked service email accounts (e.g. Evernote, Blogger, Toodledo) which create an entry when a message is sent to that inbox. The "pastlife" category is for contact information I want to hold onto, but don't really plan on actively using. I could let these go if I really needed to, but there aren't that many items in that folder.

Every now and again I import my Facebook contacts into my Gmail Contacts. To do so, you can Import your Facebook contacts into Yahoo, export from yahoo to CSV, and finally import this back into  to Google Contacts. It's kind of a pain, but I don't do it that often and allows me to keep address and phone information up-to-date, that I might not necessarily have otherwise.

It is good practice to perform maintenance on your contacts. Using the Google "merge contacts" feature and assigning new contacts into their correct folders are examples of what I do to keep things tidy. When you are done cleaning house, perform an export to get a backup of your contacts and store this file in an encrypted volume in the cloud -- more on this below! By the way this is an example of a task I have in my maintenance calendar, which recurs each year.

Gdocs
Although I mostly use dropbox to store files in the cloud, Google Docs is ideal for collaborative work.

Task Management: Maximizing Toodledo for GTD
The key for managing my individual tasks is organizing them into contexts and folders and give them each a  status, priority and sometimes a due date. I add a Toodledo iCal feed to my Google Calendars, so any tasks with dates will show up. I like Toodledo, because it syncs well with smartphone application, Pocket Informant, which I have been using since the early days with my Palm Pilot. It is my favorite mobile app interface into my tasks and also combines the Gcal information into its interface.

Toodledo Tasks

My Toodledo contexts are as follows:
  • Communications
  • Home
  • Online
  • Out & About
  • Planning
  • Tracking
  • Work
  • Workshop
  • Yard

My Toodledo folders are fairly granular and include:
  • Various names of work projects
  • Things to checkout
  • Audio
  • Call
  • Email
  • Finance
  • Habits
  • Info
  • Letter
  • Maintenance
  • Medical
  • Blog
  • Music
  • Notes (I give my notes "-1 priority", so they are not mixed in with actual tasks)
  • Personal
  • Reading
  • Remodeling
  • Spouse
  • Tech
  • Travel
  • Video

My Toodledo status fields, resemble classical GTD states:
  • Next Action
  • Active
  • Planning
  • Delegated
  • Waiting
  • Hold
  • Postponed
  • Someday
  • Canceled
I prefer to sort my tasks first by Context, then by folder, and finally using Toodledo's "automatic" setting, which means dated items first, followed by priority. Sometimes I sort by Folder first, or Priority, ToodleDo makes it easy to change how you look at your data.

I also recommend turning on the create a task via email feature. To do so click on the "Tools" menu and select "More". Then go into the "Email Import" section. Make sure the checkbox is checked, and then click the "Save Changes" button. This will enable email access to your account. Add this email address to your contacts and also save it as a auto-complete for your mobile phone so you can just type a few letters and the account will be automatically completed.

To add a task, send an email to your secret Toodledo email address with the task in the subject. In addition to the name of the task, you can also set the priority, due-date, folder, context and other values using a special syntax. For example:
Folder - To set the folder use the * symbol and then type the name of your folder.
Context - To set the context use the @ symbol and then type the name of your context.
That is all I do to customize Toodledo, and I just make sure things trickle into the structure from email, voice notes, and photos and are tagged at a minimum with Folder, Context and Status. As a side note I like to leave newly created tasks with no status, since they then appear on my mobile Pocket Informant in the inbox folder. New adds, which I've not yet categorized are also easier to find this way.

Paper & Mail
For all of my incoming paper and mail items, I carry around a plastic folder, with dividers, which I've used for many years.

My Mail and Paper Organizer 

It has the following categories, which I revise whenever I find one not being used and I need something new:

  • Inbox
  • Next Actions
  • Someday
  • To Scan
  • Waiting
  • Projects (several of these)
  • To Be Filed
I keep a separate file drawer at home, which I go through yearly to weed out old stuff. I've started to scan some things from my file drawer and organize those files into my Dropbox or Evernote folders. If it's important I'll put it into Dropbox, since I have more control over those files and can back them up, otherwise and Evernote article will work.

Evernote
I've just recently added Evernote to my bag of tricks and have not settled in yet on how I will use it. For now it has been a handy way to take pictures from my smartphone, dictate using Dragon Dictate and capture information via email, or record audio clips, which all of syncs to all of the systems I use.

I've begun storing my dream journal in Evernote and it seems like a good platform for capturing multimedia rich thought streams. I go through my Evernote notebook to process items I've captured much like I do for email. Some things I do just leave in Evernote, since it has nice organization capabilities.

I'm not ready to put my important data files into Evernote, since I like to have those in their raw data format for security and backup purposes. Although you can do a backup of your Evernote notebooks, you'd have to restore it back into Evernote and can't get directly at the files within. My Evernote setup will likely be used to get media rich information on all of my systems, which I can then properly file into a more traditional file based location (like my dropbox folders).

I'd also recommend using your Evernote email account if you like submitting things from your mobile. I have the account added to my contacts and an auto-complete for it on my mobile phone, so when I type "je" the address is auto-propagated.

1. Select a destination notebook for your email by adding @[notebook name] to the end of the subject line.
2. Add tags to your note by typing  #[tag name] at the end of the subject line. This feature works with existing tags in your account.
3. To designate a destination notebook and add tags, be sure to list the notebook name before the tags.
An example subject:
Fwd: Recipe for Bouillabaisse @Recipes #soup #fish #french

My Evernote Notebook

Dropbox
I use dropbox to organize my smaller files. I have a Western Digital My Book to store larger personal files.

Here are the folders for my Dropbox accounts, which I try to keep fairly flat and unnested:
  • _safe (contains the following TrueCrypt encrypted volumes: data.tc, small.tc, journals.tc, pim.tc)
  • astrology
  • carving
  • consulting
  • cygwin
  • fitness
  • for sale
  • genealogy
  • shopping
  • insurance
  • jobsearch
  • letters
  • lists
  • network
  • recipes
  • papers
  • programming
  • spiritual
  • templates-forms-spreadsheets
  • travel
  • web
  • work
Encryption is important to me for the safety of my data. I keep a large TrueCrypt volume with data that rarely changes since the entire volume has to sync if you make a change. Within the "_safe" folder I have a small volume, which contains spreadsheets with passwords and so forth.  A third TrueCrypt volume is used for private journals.

To use TrueCrypt in with Dropbox it is important to change the TrueCrypt setting to allow the synchronization of a volume, which will never change its byte size, but will have a new date. The change you need to make is to uncheck "preserve modification timestamp of file containers" preference.

And here are my WD MyBook folders, which have pretty deep nesting and many sub-categories for fine grained separation of a lot of material. By the way the WD MyBook is a network appliance which I can access from all the systems behind my wireless router. I have a reminder in my maintenance calendar to make a backup of this data quarterly, along with what I store in the cloud.

My large volume folders on my personal network share quickly go from broad to specific topics and I've made great effort to continually reduce categories to the minimum needed and creating additional  sub-folders where I want a better delineation of topics.
  • audio
  • binaries
  • print-media
  • videos
Bookmarks
As I like to take in a lot of information on the web, it is important that I organize and update my bookmarks. I use Xmarks for bookmark synchronization across all multiple systems, platforms and browsers. I primarily employ Chrome for browsing, IE for a few banking sites, and Firefox for its bookmark syncing application that runs on my mobile device.

Another thing I like to do is sort all of my bookmarks. I do this in Firefox using Andy Halford's sort places add-on.


I keep my bookmarks within a hierarchy on the bookmarks toolbar. Here is what that that top-level hierarchy looks like:


Code
For storing my code, I use two services, which offer free versioning services.
Mobile Tools
These are the tools I use on my iPhone to look into the world above as well as channel information into the tools I've mentioned thus far. The only things I can't get to at the moment, are my TrueCrypt encrypted volumes.
  • Dragon Dictate
    • Create iPhone abbreviations, to your automated email addresses for services like Toodledo, Evernote, and Gmail, as well as your personal email. That way you can dictate and send to those addresses fairly easily. (I don't have Siri!)
  • Firefox Home (This allows me to get to my latest bookmarks. You can set this app to use Safari as the default browser.)

Tuesday, December 13, 2011

My Favorite Geek Documentary Films

When my wife goes out of town I like to watch geek documentaries. I'd like to share some of my favorites, which are freely available for streaming. Someday I'd like to be a producer!

Yes, that is Parmesan.

Some of the best sites for watching free documentaries are:

And if you subscribe to Netflix streaming and have ever worked in computer support, check out this British comedy-drama for some good laughs:

Friday, December 9, 2011

Amit Rathore's Introduction to Clojure: Week 1

I'm currently taking the Introduction to Clojure class from CodeLesson.com taught by Amit Rathore, author of Clojure In Action. (You may be able to get a 40% discount on the book if you use "clojure40" as the coupon code when purchasing through Manning, I'm not sure how long this offer is good for.) We are using the book for our coursework material along with Amit's quizzes and exercises. There are 14 students in the class, 9 from the US and the others are from Germany, Spain,  Ireland and the UK.

For the first week we've read through chapters 1 - 3 of Clojure in Action, taken 3 quizzes and have been given, what has been for me a challenging exercise. The class site uses moodle, unfortunately there has not been any audio/video presentations yet. I'm still hoping.

This week's exercise gives us a string of inputs consisting of a matrix size, [x y] coordinates and heading of an object, along with a series of left and right turns and forward movements for the object. The end goal is to have the output reveal the proper position of the object in the matrix. For extra credit we are encouraged to add collision and boundary detection, which I've not done. Credit goes to my classmate John Vieten for showing me how this was done. I copied John's design and re-implemented based on that, although I had to look back at his code from time to time as I'm still having difficulties thinking functionally. Also check out Amit's version of the solution.

;; had some frustrations with things needing to be (str)'ed
;; thus all the debug statements...

(def left-bearing-lookup {"N" "W", "W" "S", "S" "E", "E" "N"})
(def right-bearing-lookup {"N" "E", "E" "S", "S" "W", "W" "N"})

(defn turn-left [bearing] 
  (do
    ;(println "LEFT:" bearing (left-bearing-lookup bearing))
    (left-bearing-lookup bearing)))

(defn turn-right [bearing] 
  (do  
    ;(println "RIGHT:" (right-bearing-lookup bearing))
    (right-bearing-lookup bearing)))

(defn move-rover [matrix]
  (let [bearing (:bearing matrix)
        x (:x matrix)
        y (:y matrix)]
    ;(println "MOVE-ROVER:" matrix bearing x y)
    (cond
      (= bearing "N") (assoc matrix :y (inc y)) 
      (= bearing "S") (assoc matrix :y (dec y)) 
      (= bearing "W") (assoc matrix :x (dec x)) 
      (= bearing "E") (assoc matrix :x (inc x))))) 
      
(defn parse-moves [matrix move]
  ;(println "PARSE-MOVES-entry:" move)
  (cond 
    (= move \L) (assoc matrix :bearing (turn-left (:bearing matrix)))
    (= move \R) (assoc matrix :bearing (turn-right (:bearing matrix)))
    (= move \M) (move-rover matrix)))

(defn parse-rovers [matrix [[x _ y _ bearing] moves]]
  (let [x (Integer/parseInt (str x))
        y (Integer/parseInt (str y))
        bearing (str bearing)
        matrix (assoc matrix :x x :y y :bearing bearing)
        new-matrix (reduce parse-moves matrix moves)
        ]
    (do
      ;(println "BEARING:"(:bearing matrix))
      ;(println "ORIG:"matrix)
      ;(println "NEW:"new-matrix)
      (println "[" (:x new-matrix) (:y new-matrix) (:bearing new-matrix)  "]"))))

(defn process-lines [rest-lines]
  (let [rover (take 2 rest-lines)
        rover-array rover]
    (if-not (= rover [])
      (cons rover-array (process-lines (nthnext rest-lines 2))))))

(defn run-rovers [input]
  (let [lines (.split input "\n")
        [x y] (.split (first lines) " ")
        x (Integer/parseInt x)
        y (Integer/parseInt y)
        rest-lines (rest lines)
        ;;reduce is the bomb!
        matrix (reduce (fn [mymap keyvec] (assoc mymap keyvec :bearing)) 
                      {} 
                      (for [x (range 1 (+ x 1)) 
                            y (range 1 (+ y 1))] 
                        [x y] ))]
    ;(println matrix)
    ;(println rest-lines)
    ;(println (process-lines rest-lines))
    (reduce parse-rovers matrix (process-lines rest-lines))))

(run-rovers "5 5\n1 2 N\nLMLMLMLMM\n3 3 E\nMMRMMRMRRM")

Mostly for my own review, here is a brief overview of the first 3 chapters, although the actual chapters cover  A LOT more than what I glance over. While you'll need to check out the Clojure in Action book yourself to experience the expansiveness and ease of  the material presented, I highlight some of the topics, which I sometimes hadn't quite understood from reading other books. I've added links to ClojureDocs where applicable.

Chapter 1: This chapter covered a general overview of the language and its capabilities diving into concepts here and there that would later be highlighted. I really enjoyed the breakdown of the Clojure runtime versus the stages of a typical language processor and the chart showing Clojure compared with other languages, which I've taken the liberty of including:

Figure 1.4 from Clojure in Action (pg. 26)

Chapter 2: Covers the whirlwind language tour. Similar in content and style to chapter 2 "Drinking from the Clojure firehose" from the Joy of Clojure book, this chapter familiarizes the reader with the basics of the language and introduces idiomatic ways of doing things using special, forms, functions and macros.

loop: (loop bindings & body)

Much like let the loop/recur form is used by specify binding within the loop function call which are then replaced by the recur call. This operation does not consume the stack but recur can only be used in the tail position, which simply means it has to be the last statement in the loop/recur form.

(defn factorial-loop [n]
  (loop [current n factorial 1]
    (if (= current 1)
      factorial 
      (recur (dec current) (* factorial current)))))

doseq, dotimes

These forms take a 2 term vector, the first is a new symbol (variable), which is then sequentially bound to each element in the the sequence from the second term. Doseq works with a function and dotimes works with a number as the second term.

map

Takes a function and a sequence of data which is then applied to that function returning a new sequence. If you give it multiple functions you have to have an equal number of sequences.

filter

Accepts a predicate, which is something that returns true or false, and a sequence which then only returns those elements that return true for the predicate.

reduce

A function which which accepts a function that expects 2 terms and a sequence of elements. The first two elements are pulled off and processed by the function, then the result of this call plus the next function are called on the function until the sequence has been gone through.

(defn factorial-reduce [n]
  (let [num (range 1 (+ n 1))]
    (reduce * num)))

for: (for seq-expr body-expr)

Used for list comprehension and can be further refined using :let, :when, and :while keywords.

(defn matrix [x y]
     (for [x (range 1 (+ 1 x)) y (range 1 (+ 1 y))] [x y]))

Which results in:

=> (matrix 3 3)
([1 1] [1 2] [1 3] [2 1] [2 2] [2 3] [3 1] [3 2] [3 3])

thread-first ->, thread-last ->> 

Macros to make nested function definitions easier to read.

assoc-in, get-in, update-in

Functions which make it easy to dig into nested maps.

Chapter 3: Dives into functions, which I'll cover a bit and scope and destructuring, which I won't get into. I found the full structure of the defn macro quite helpful:

(defn function-name doc-string? attr-map? [parameter-list]
  conditions-map?
    (expressions))

The conditions map can take :pre and :post keywords which are applied before and after the function is executed.

arity

This is the number of arguments a function can take. A function which takes variable arity is said to be variadic and uses the form (defn function-name [& nums] (apply + nums)). You can precede the & with a known number of symbols if it has to have a certain minimum arguments, the remainder will be returned as a single list.

overloading

In order to overload a function you supply multiple arity parameters with bodies within a single function.

trampoline

This function allows you to call mutually recursive functions which do not eat up the stack. An example would be one recursive function calling another and the other calling the first. You have to use (declare function-name) for the second function, which is used in the first function, since it won't exist the first rime through.

time

Wrap your function calls with time to get the elapsed time taken for execution.

memoize

Wrap your function with a call to memoize, which will cache the results upon the second call. Can be used to speed things up in certain scenarios.

lexical closures
(the above link takes you Andrew Buntine's blog, which has a nice, simple explanation of the concept)

These are forms that enclose over free variables, which means the variable has no binding within the lexical scope of the form.
(defn make-scale [scale]
  (fn [x]
    (* x scale)))

(def make-percent (make-scale 100))

(make-percent 1.20)

Which results in:
=> (make-percent 1.20)
120.0

metadata

Can be used much like annotations.

Free Online Stanford iPhone IOS5 Development Course


Stanford offers a free online beginning iPhone Development course.
The lectures were updated this month for ios5 and are available through iTunes at no cost.

Course Description:

Updated for iOS 5. Tools and APIs required to build applications for the iPhone and iPad platform using the iOS SDK. User interface designs for mobile devices and unique user interactions using multi-touch technologies. Object-oriented design using model-view-controller paradigm, memory management, Objective-C programming language.

Other topics include: object-oriented database API, animation, multi-threading and performance considerations.

Prerequisites: C language and programming experience at the level of 106B Programming Abstractions (also offered at no cost, I checked out a couple of these a few years ago and they are great) or X. (FYI - 106A is an entry level Java course with a great, enthusiastic instructor)

Recommended: UNIX, object-oriented programming, graphical toolkits Offered by Stanford’s School of Engineering, the course will last ten weeks and include both the lecture videos and PDF documents. A new lecture will be posted each Monday, Wednesday and Friday. Subscribe to this course, and automatically receive new lectures as they become available. Released with a Creative Commons BY-NC-ND license.

Thursday, December 1, 2011

Making Eclipse, CounterClockWise and Leiningen Sing on Windows

Download the Leiningen batch file for Windows to get started. You'll likely need to edit the file to set the path for where you are at.

set LEIN_HOME=c:\dev\leiningen

Next you need to run the Leiningen installer:

lein.bat self-install

If you are behind a firewall you can download the standalone jar file, place it in the 'self-installs' directory. That way you won't need to mess with curl or wget.

The rest of Leiningen uses maven, so this is how you get around an http proxy with maven. Create a setting.xml file in your user's home directory within a folder called .m2.

C:\Users\joe_user\.m2\settings.xml:


<settings>
  <proxies>
    <proxy>
      <id>default</id>
      <active>true</active>
      <protocol>http</protocol>
      <host>PROXY_HOST</host>
      <port>PROXY_PORT</port>
 <username>YOUR_PROXY_USERNAME</username>
      <password>YOUR_PROXY_PASSWORD</password>
      <nonProxyHosts>localhost</nonProxyHosts>
    </proxy>
  </proxies>
</settings>

At this point I'd recommend testing things out from the command line.

Test line.bat to ensure you get the help menu.

Create a new project using lein.bat new.

Pull in dependency's for the project by going into the project directory and running lein.bat deps.
This will test the maven proxy configuration.

Once you have Leiningen setup and installed it's time to setup eclipse. I'm going to refer to the labrepl wiki instructions for setting up Eclipse with counterclockwise. You don't have to go all the way through the instructions since we aren't setting up labrepl, but it wouldn't hurt to install the Git and Maven plugins as well as CounterClockWise, which are all discussed.

Great, now I'm going to refer you to the sexp.posterous blog, which has simple instructions to create the external calls to the Leiningen script from Eclipse. These can then be used by all of your Clojure projects and should only take you about 5 minutes to setup.

Now you have your basic Eclipse-Clojure-Leiningen environment setup and I'll cover general use and some gotchas that I ran into.

The first thing you want to do is create a new Eclipse project. Select the 'Clojure' type for your project. If you don't see this option, then perhaps you didn't get the CounterClockWise plug-in properly installed. It should look like this:

New Project Wizard

We will now run some of the Leiningen external tools you setup earlier. If you have difficulty getting the commands to run, I've found I sometimes have to click on the project name before I run the command, as I'm showing in this pic:

External Tools Drop-Down

Run the Leiningen 'new' command first. This will add some files and folders to the existing Eclipse project. (You may need to refresh the project before they appear.) The files of interest are the project.clj file in the project root and the core.clj files under /src and /test in your project's namespace, which in my case is helloWorld - the same name as the project. It get's this name, based on how we setup the Leiningen 'new' command. You can think of the Clojure namespace being similar to a Java package.

Your setup should now look something like this view from the Eclipse package manager:

Package Manager View

Next you will run the Leiningen 'deps' command from the external tools drop-down. This will look at the project.clj file and download any necessary dependencies and put them into the /lib folder. Leiningen uses  Maven to do this, which is why I had to add the settings.xml file to Maven to get through my http proxy. You will not need to do this if you don't have to deal with a proxy.

Now let's add the test folder that was created by the Leiningen 'new' command to the project's source build path in Eclipse:

Java Build Path Window

You'll notice that after running the 'dep' command you have a build error on your project. This is because 'dep' wipes out the /classes folder. 

Run the Leiningen 'compile' command and it will be restored. You may need to refresh your project after you run compile. 

FYI: Just so you know when you are in the package manage view, if there ever is a folder that you know exists, which you cannot see, it is likely because the folder is configured as a build path library object for your project.

Now let's modify the default code and create a simple hello world program which we want to be able to package and run outside of Eclipse. To do this you will modify the src/core.clj and project.clj files.

Aside from the hello world function, you will also need to add the :gen-class specification to your core.clj file in order to create runnable java class files. Note that we have given the function the name of -main, which is the same as the main method for a Java program.

(ns helloWorld.core
(:gen-class))

(defn -main [msg]
(println "Hello" msg))

Now on to the project.clj file, which lists dependencies and also needs to point to the main method.

Note the :manifest specification in the project.clj file shown below. The manifest does not need to be specified if you are going to only use the standalone jar using Leiningen's 'uberjar' tool, since clojure is included with standalone jar. If however, you wanted to have several projects deployed somewhere which reference only one instance of the clojure library to save space, you can specify where your libraries live using :manifest.

(defproject helloWorld "1.0.0-SNAPSHOT"
:description "FIXME: write description"
:dependencies [[org.clojure/clojure "1.3.0"]]
:main helloWorld.core
:manifest {"Class-Path" "lib/clojure-1.3.0.jar"})

Now run the 'uberjar' command, which will create 2 jar files of your project. One which includes the clojure library, it will have STANDALONE in its name, and another jar which merely references your clojure library, which was specified using :manifest.

You should be able to run your project like so:


c:\UL\workspace-helios32\helloWorld>java -jar helloWorld-1.0.0-SNAPSHOT-standalone.jar Joshua
Hello Joshua


If you don't give this program any parameters it will throw the following "arity" exception. Maybe more about that another time... Enjoy your new environment!


c:\UL\workspace-helios32\helloWorld>java -jar helloWorld-1.0.0-SNAPSHOT-standalone.jar
Exception in thread "main" clojure.lang.ArityException: Wrong number of args (0)
 passed to: core$-main
        at clojure.lang.AFn.throwArity(AFn.java:437)
        at clojure.lang.AFn.invoke(AFn.java:35)
        at clojure.lang.AFn.applyToHelper(AFn.java:159)
        at clojure.lang.AFn.applyTo(AFn.java:151)
        at helloWorld.core.main(Unknown Source)

My Clojure Reading List


Here are the current Clojure books on the market. I've got all of these on my "to-read" list. I started "Joy of Clojure", which I really liked and can't wait to better understand, but had to rewind to Halloway's "Programming Clojure", which is geared for a beginner without previous Lisp experience.


Good for Beginner's:

Intermediate to Advanced:

And Coming soon: