301 Moved Permanently

http://groovecoder.com

quick blurb on NoSQL

I've spent about as much time thinking about this as NoSQL developers spend thinking about their schema, but here it is anyway.

At SourceForge I'm presently developing and maintaining a few different systems using all kinds of web tech's and languages - PHP, python, solr, Postgres, MySQL, and mongo. One thing I'm noticing is that the mongo systems are something of a breeze to write, and then a real challenge to maintain - especially debugging. Our mongo experts mostly say that the tooling for mongo is just 'immature.' I'm sure they're right, but that also points toward what I think might be a fundamental difference in the two modes of development.

AFAIK, there aren't any "old" NoSQL systems around? Mongo is only out since 2007, and Cassandra since 2008? We started using mongo early 2009, and even just one year out it feels so much more painful to maintain than our Postgres or MySQL systems that have been around since 1999! My theory is that NoSQL sacrifices maintenance and future development effort for the sake of startup development. I even made a neat drawing:



Initially mongo seems to save on effort until the first valley - initial launch. At this point, the system launches and typically starts interacting with other systems and with users - data requirements change towards reality, which means code - i.e., function and logic - changes, not just model. In our environment, all other systems that use the data must also change their code which seems harder than the originating code. The code and the data are so intermixed that seemingly any and every change in either domain makes knock-on effects that have to be addressed.

In a typical schema data system, we front-load a bit of data modeling effort. After launch, when we get new and changing data requirements we typically address the schema changes that might be involved, and may have to write a data migration/transformation script. But beyond that, it seems we don't have to worry about data integrity or any other knock-on effects. We can change some data-access or model classes and be on our way.

So, am I just an old crusty developer shouting at these NoSQL kids to "get off my lawn!" ? Or has anyone else noticed this too? Maybe it's just the heterogeneous mix of NoSQL + schema that's killing me. Just seems like such a pain for not enough benefit?

a rant about ranting

Disclaimer: this post is totally my own opinion and does not reflect anything from SourceForge at all. that's why it's here on this blog.

I'm angry and want to shoot my mouth off - perfect opportunity for a long-lost blog.

We - i.e., SourceForge are getting some crap for blocking sanctioned countries from our site. That's fine - I'm actually ticked off about it too. And many people out there are making sound and solid comments about the action - not just the ones defending SourceForge; there are some good solid critical comments too.

But then you have people who say something like this:

Sourceforge, you suck! You suck so badly, I’ll hereby guarantee you that I’ll not only recommend *anybody* stay the heck away from you scumbags, I’ll actively let everybody know that you’re the scum of the earth. Shame on you! Shame!

With love from pyalot. Well pyalot, since we're all good to judge and criticize each other, let's get started ...

So you are Florian Bösch. Okay Florian, let's see here ... you've worked at Systor(?), Accenture, and DWS. Systor doesn't seem too keen on open-source?, nor does DWS. Ah, looks like Accenture has some good open source work; but what's this?! It's right alongside Microsoft and Oracle solutions?! OMFG! You are the scum of the earth for working with them! GRARRR!

Or, if I take an extra minute, I find you're actually a stand-up guy and developer and a good contributor to open-source!

Couple lessons here:

1. we're not anonymous on the internet anymore; I found all of this info on Florian starting from his sf.net user page


2. when we only look at a single facet of any news story or party, we get a very distorted view

I actually sympathize with Florian's sentiments - blocking access from countries goes against the FLOSS ideal. But at the end of the day, SourceForge is a US company under US law. And if we're not law experts we should probably speak our opinion quietly or not at all.

OSCON quotes - day 1

I want to share quotes I overhear at OSCON 2009. Most of these are from fellow SourceForgers ...

• I'm a fan of the minimalist beauty of the electronic device.


• Your API is not a beautiful fucking snowflake.


• I am as asymptotically close to clean as possible.


• You're going to be happy about not being happy.


• I'm German, we know how to deal with crowds.


• It doesn't matter, you eat it with rice and bread.


• I fucked the grower to get this shit.


• It's amazing what you can fit up your ass with a little practice.


• I don't like my balls soaked in sugar syrup.


• People shouldn't call each other tar pit.


• There's nothing you can think of with an olive that I haven't already video'd and sold on the internet.


• Is this the placenta thing?


• All eating human flesh stories start with, "I was going to med school."

Test-Driven [Design|Development]

Today I learned to appreciate Test-Driven Design a little bit more. Here's the story.

I'm writing some RSS feeds that will contain extensions and other non-RSS elements using XML Namespaces. I'm using Zend_View and Zend_Feed and I thought the best place to put the namespace would of course be at the top of my default.rss.phtml template file - that way I can register all the namespaces at once at the top of the feed. Instead of writing the test first, I wrote the code first. Took maybe 10-20m and seems to work fine:

<rss content="http://purl.org/rss/1.0/modules/content/" 
doap="http://usefulinc.com/ns/doap#" 
sf="http://sf.net/api/sfelements.rdf#" 
foaf="http://xmlns.com/foaf/0.1/" 
rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
version="2.0">
....
</rss>

Then I go to write the test. Lo and behold - it's a big pain in the ass to consume the feed using SimpleXML.

It's easy enough to create a SimpleXML element out of the feed, but I can't create SimpleXML elements from the content:encoded XML data:

<content:encoded>
<!--[CDATA[<doap:version>
<doap:name>Project 1.1 - Foobaj</doap:name>
<doap:created>1202221896</doap:created>
<doap:helper>
<foaf:person>
<foaf:name>admin1</foaf:name>
<foaf:homepage resource="http://lcrouch-703.sb.sf.net/users/admin1">
<foaf:mbox_sha1sum>6dd817a0f71590a68131a5e83b1bd73944654e8d</foaf:mbox_sha1sum>
</foaf:Person>
</doap:helper>
<doap:file-release>proj1.file1.tgz</doap:file-release>
<sf:download-count>0</sf:download-count>
</doap:Version>]]-->
</content:encoded>

Because all the namespaces used in the DOAP class aren't in the content. Argh! My first thought is to screw SimpleXML and do a raw string search/parse in the test. But then I had my epiphany: "If I were an actual client of this feed, I would want to be able to parse it easily with SimpleXML or with any other XML library."

I ended up pushing the xml namespace declarations right down into the appropriate elements - where I now think they are *supposed* to be:

<content:encoded>
<!--[CDATA[<doap:version 
doap="http://usefulinc.com/ns/doap#" 
sf="http://lcrouch-703.sb.sf.net/api/sfelements.rdf#">
<doap:name>Project 1.1 - Foobaj</doap:name>
<doap:created>1202221896</doap:created>
<doap:helper>
<foaf:person 
foaf="http://xmlns.com/foaf/0.1/" 
rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<foaf:name>admin1</foaf:name>
<foaf:homepage resource="http://lcrouch-703.sb.sf.net/users/admin1">
<foaf:mbox_sha1sum>6dd817a0f71590a68131a5e83b1bd73944654e8d</foaf:mbox_sha1sum>
</foaf:Person>
</doap:helper>
<doap:file-release>proj1.file1.tgz</doap:file-release>
<sf:download-count>0</sf:download-count>
</doap:Version>]]-->
</content:encoded>

Voila - SimpleXML starts parsing everything very easily.

This is one of the biggest boons for Test-Driven Development - the effects it has on the way you design your code. If I had not tested my code as an actual client would use it, I would have produced some pretty shoddy feeds with useless XML namespacing.

Leave the editor open

I've been trying to adopt some practices from The Productive Programmer. Mostly by using more keyboard shortcuts and productivity tools like Quicksilver, Jumpcut, etc.

Yesterday and today I realized a productivity tactic that isn't in the book - just leave your work open when you "go home" for the night. Don't close the program. In fact, don't even close any files, tabs, or any background programs either. Just save everything and walk away.

The effectiveness of this trick is related to something Joel wrote about a while back ...

For me, just getting started is the only hard thing. An object at rest tends to remain at rest. There's something incredible heavy in my brain that is extremely hard to get up to speed, but once it's rolling at full speed, it takes no effort to keep it going. Like a bicycle decked out for a cross-country, self-supported bike trip -- when you first start riding a bike with all that gear, it's hard to believe how much work it takes to get rolling, but once you are rolling, it feels just as easy as riding a bike without any gear.
Maybe this is the key to productivity: just getting started. Maybe when pair programming works it works because when you schedule a pair programming session with your buddy, you force each other to get started.

In the bike metaphor, leaving all your work open is like leaving the bike poised on a down-hill slope. All you have to do is get back to it and hop on. If I sit down at a blank desktop, I'm more likely to open my email, read my RSS feeds, open work email, and THEN, finally, open my code editors. If I sit down in front of a code editor, I'm likely to start editing code immediately.

Seven things that probably you may not know about me

Anderson tagged me, so I'll give this a try, though I'm going to have a tough time finding 7 other people who haven't been tagged already.

• I have a black belt in the hodge-podge kick-boxing-jujutsu-taekwondo-karate style of fighting they teach at Apollo's Karate.


• I have an identical twin brother, and 2 older brothers, one of whom is also a PHP developer.


• I am emerging Catholic.


• I brew my own beer.


• I love soccer. I try to play every weekend. Also, GO REDS!


• I can speak conversational Russian. I also speak a little French, a tiny bit of German and Portuguese, and I'm starting to learn Spanish. I'm only fluent in English though. :(


• I landed my job at SourceForge after I made an OSS project there. So go make one yourself! :)

I'll tag ...

• Matt Crouch - above-mentioned brother.


• Travis West - SourceForge colleague and long-time friend.


• Steven Osborn - developer from Vidoop who helped me run the Tulsa PHP User Group until he ditched us for Portland. ;)


• Noah Everett - Tulsa PHP developer I met thru TPUG; he created and maintains twitpic.


• Brad Vernon - Tulsa PHP + Ruby developer I met thru TPUG.


• Vance Lucas - Another PHP developer here in Oklahoma. Met him at Tulsa Tech Fest 2008.


• Rafael Dohms - PHP developer in Brazil; he saved my ass at PHP Conference Brasil '08 when he found a DVI-VGA adapter for me to present my keynote.

So tagging it back to Brazil, from whence I was tagged. :)

Unit-testing ZF Controllers without Zend_Test

I've read a couple articles and blog posts recently talking about Zend_Test and/or testing Zend Framework Controllers. Particularly for controller testing, I'm kinda surprised how much plumbing code people are using. I recently started testing some Zend_Controller code (from ZF 1.5 even!) at SourceForge and did not do nearly that much plumbing.

Basically, I want to test the controller code in isolation from the front controller, the router, the dispatcher, the views, etc. All I to do is set up a request object, invoke the action methods of the controllers, and then assert against the variables assigned to the view. For these tests, I don't care about the output of the view templates themselves - I just want to know the controllers are putting the right variables into the view object.

It turns out this is actually pretty simple. I made a custom test case:

class Sfx_Controller_TestCase extends Sfx_TestCase
{
    protected $_request;
    protected $_response;
    protected $_controller;
    
    public function setUp()
    {
        parent::setUp();
                
        // set up smarty view and restful view helper
        $viewRenderer = new Sfx_Controller_Action_Helper_TestViewRenderer();
        Zend_Controller_Action_HelperBroker::addHelper($viewRenderer);

        $this->_request = new Zend_Controller_Request_Http();
        $this->_response = new Zend_Controller_Response_Cli();

    }
}

Sfx_TestCase contains all my bootstrap code. However, the only thing I do in bootstrap is set include path and set up a default db adapter for Zend_Db_Table. I don't do anything with Zend_Controller_Front. So this may as well extend straight from PHPUnit_Framework_TestCase. I'm not sure why others are claiming you have to use Zend_Controller_Front to test ZF Controllers - you don't.

I wrote and use Sfx_Controller_Action_Helper_TestViewRenderer (and proposed it as a core class) to simply create an empty Zend_View object into which the controllers can assign variables. Here's the whole class:

class Sfx_Controller_Action_Helper_TestViewRenderer extends Zend_Controller_Action_Helper_ViewRenderer
{
    public function initView()
    {
        if (null === $this->view) {
            $this->setView(new Zend_View());
        }
        // Register view with action controller (unless already registered)
        if ((null !== $this->_actionController) && (null === $this->_actionController->view)) {
            $this->_actionController->view       = $this->view;
        }
    }
}

With only this much plumbing, I'm able to test the Controllers in isolation - no worrying about routes, dispatchers, plugins, helpers, nor view templates - like so:

class ProjectControllerTest extends Sfx_Controller_TestCase
{    
    private function __constructProjectController()
    {
     return new ProjectController($this->_request, $this->_response);
    }

    public function test_indexAction_fetches_all_projects()
    {
        $this->_controller = $this->__constructProjectController();
        $this->_controller->indexAction(); // assigns 'resources' to view
        $this->assertNotNull($this->_controller->view->resources);
        $this->assertEquals(27,count($this->_controller->view->resources));
    }

    public function test_indexAction_new_since_fetches_only_new_projects()
    {
        $this->_request->setParam('new_since',1205880839);
        $this->_controller = $this->__constructProjectController();
        $this->_controller->indexAction();
        $projects = $this->_controller->view->resources;
        $this->assertEquals(4,count($projects));
        foreach($projects as $project){
            $this->assertGreaterThan(1205880839, $project->create_time);
        }
    }

    public function test_indexAction_limit_limits_projects()
    {
        $this->_request->setParam('changed_since', 1205880839);
        $this->_request->setParam('order_by','changed_since');
        $this->_request->setParam('limit', 5);
        $this->_controller = $this->__constructProjectController();
        $this->_controller->indexAction();
        $projects = $this->_controller->view->resources;
        $this->assertEquals(5,count($projects));
        $prevChangeTime = 0;
        foreach($projects as $project){
            $this->assertGreaterThanOrEqual($prevRegTime, $project->change_time);
            $prevChangeTime = $project->change_time;
        }
    }

}

I'm finding this to be a much simpler and easier way of testing ZF Controllers than the other articles I've been reading. Now if you want to test everything in the front controller dispatch process and the view templates, I think Zend_Test is the best bet, but I've not used it yet so I can't be sure. The above classes work fine for what I do.

Kiva.org

Kiva.org is a remarkable organization in at least a couple ways - they employ micro-finance principles to aid entrepreneurs in developing countries, and they make excellent use of online technology to do so. We highlighted them in my International Aid and Development class in college.

I really like that they're using the Long Tail on both the lending side and the receiving side of micro-finance. I also like some of their cool web features - the portable badge above, and their use of Facebook Connect to syndicate their activity to Facebook.

One of our good friends gave us a $25 gift certificate to Kiva and I think it's one of the best gifts we've ever received. I've admired Kiva for a while but have never spent the time or effort to get involved with it; this small amount is really inspiring me to do more.

UPDATE: Wow. When I picked my loan recipient, Margaret, she had 0% of her requested loan. In the 1-2 hours it took to get this blog post up, she received 100% of it. Go Kiva! Go Margaret!

ZF Rest classes

Holy crap. I forgot that way back after Tulsa Tech Fest I promised to upload some Zend_Controller classes I wrote to enable RESTful behavior. I think the presentation did a decent job conveying their purpose and operation, so here, finally, are the classes themselves.

If anyone does go look at them, feel free to comment/question here on this post about them.

PHP Brasil '08

I have posted a trip report about PHP Brasil '08 over at the SourceForge.net Community Hub. There's also a video of my talk, a link to Chris Jones's thorough trip report, and links to my presentation slides.

Laughably Ridiculous

Okay, although I'm an open-source devotee, I've actually intellectually bantered in favor of copyright law. I know, I'm sorry; but I can understand the philosophical underpinning of *a* copyright scheme ... even if I don't agree with its effectiveness.

But this is just getting absurd.

Interestingly, SPFF is also going after Sourceforge, the open source development website, because it hosts the P2P application Shareaza.

So let me state this matter-of-factly:

In suing SourceForge, SPFF is not suing an entity who distributes copyrighted material. They're not even suing someone who develops software that might be used to distribute copyrighted material. SPFF is suing someone (i.e., SourceForge) who develops software (i.e., sf.net) that might be used to develop or distribute software (i.e., Shareaza) that might be used to distributed copyrighted material.

Oklahoma State Question 743

Here is, verbatim, text I received from the local homebrew shop about Oklahoma State Question 743. I think it's important for people to be informed when they vote, so I'm passing this along ... obviously it's biased pro-wine-makers...

Fellow Oklahoma winemakers and homebrewers,
Tomorrow's election will have a state question that has a huge impact on the Oklahoma winemaking industry. In 2001, voters in Oklahoma voted over 70% in favor of allowing Oklahoma wineries to sell directly to liquor stores and restaurants without going through a distributor. That change allowed the wine industry to go from a few wineries to over 50 in just a few short years. Small wineries were able to sell to the local liquor stores and restaurants without being at the mercy of a wholesaler that had little interest in distributing for every little winery that opened here in Oklahoma.

Last year this law was challenged by the distributors as unconstitutional and was overturned by the state supreme court. The reasoning was that it created an unfair advantage for Oklahoma wineries over out of state wineries who were still required to go through a distributor. It was a huge blow. Out of state wineries that distribute in Oklahoma would more than likely use a distributor regardless.

In order to make the law "fair", a new question will be on tomorrows ballet. It rewrites the law to include any winery that produces under 10,000 gallons a year. Oklahoma wineries are dying on the vine right now. With out this change many will not succeed. Please pass this on to friends so that we can ensure that this law passes.

The following is the actual question appearing on the ballot:
State Question 743 - In Short: Wineries from Oklahoma and outside the state of Oklahoma will be able to sell their wine directly to retail stores and restaurants if SQ 743 is approved. Currently, they can only do so through a wholesaler or at fairs/festivals.

Actual Ballot Text:

This measure amends Section 3 of Article 28 of the Constitution. It requires a customer to be twenty-one and physically present to purchase wine at a winery, festival or trade show. The measure changes the law to allow certain winemakers to sell directly to retail package stores and restaurants in Oklahoma. The change applies to winemakers who produce up to ten thousand gallons of wine a year. It applies to winemakers in state and out of state. Those winemakers may not also use a licensed wholesale distributor. They must sell their wine to every retail package store and restaurant in Oklahoma that wants to buy the wine. The sales must be on the same price basis. The sales must be without discrimination. Those winemakers must use their own leased or owned vehicles to distribute their wine. They may not use common or private carriers. If any part of this measure is found to be unconstitutional, no winemaker could sell wine directly to retail package stores or restaurants in Oklahoma.

Designing Simplicity

Dang! I wish I had this quote when I was making my REST slides for Tulsa Tech Fest. It's perfect!

I think most people just make the mistake that it should be simple to design simple things. In reality, the effort required to design something is inversely proportional to the simplicity of the result.
-Roy Fielding

The man himself. Priceless. Goes along great with the one I did use:

A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away.
-Antoine de Saint-Exupry

Framework Performance according to Rasmus

Alright, so invoking Rasmus in the title is a bit provocative, but I stumbled on an interesting talk of his; showing performance benchmarks for a number of popular php frameworks. The first portion of the talk is exactly what he presented @ OSCON, but the second half looks to be raw performance numbers. It looks like there are a couple specific, but easy, performance tweaks that he granted to certain frameworks, and I like seeing the data so much I thought I could try my hand at distilling it into Google Docs charts.





It's interesting that some tulsaphp guys and were recently talking about Zend had to be the slowest of all the frameworks, but apparently not so! That honor goes to CakePHP?

Tulsa Tech Fest 2008 Slides

As promised, I'm posting my slides from my Tulsa Tech Fest 2008 presentation - RESTful MVC in Zend Framework.

the binary canary testing pattern

I think I just invented a new testing pattern - The Binary Canary.

Basically, I was grouping my PHPUnit tests into a test suite and I realized that my TestCase super-classes were "failing" because they had no tests in them. Obviously this is intentional - only the specific sub-classes would have tests.

I guess I could have made the TestCase super-classes abstract, but instead I added this to the highest-level TestCase class:

/*
 * global test plumbing here
 */
class Sfx_TestCase extends PHPUnit_Framework_TestCase
{
 public function setUp()
 {
  // more global test plumbing here
 }
 public function test_Binary_Canary()
 {
  $this->assertEquals(
         "Binary Canary says test plumbing is working.", 
         "Binary Canary says test plumbing is working."
        );
 }
}

My little binary canary serves two purposes:

1. It adds an "always-pass" test to each of my TestCase classes so they don't throw up any more PHPUnit warnings.


2. Because my TestCase classes set up context-specific test plumbing, the binary canary test inherited by each of them now alerts me if I screw up any of my test plumbing - and tells me the specific area.

For example:

class Sfx_Db_TestCase extends Sfx_TestCase
{
 public function setUp()
 {
  parent::setUp();
  // Db-specific test plumbing
 }
}

And:

class Sfx_Controller_TestCase extends Sfx_TestCase
{
 public function setUp()
 {
  parent::setUp();
  // Controller-specific test plumbing
 }
}

Just like the coal-miner canaries of old, this mechanism gives me a simple yes/no signal as to whether or not my test plumbing will soon kill me, and which plumbing code is the culprit.

bailouts

I usually try to avoid preachy blog postings, but I can't help it today. Blame it on "casual Friday" or something.

Like lots of people, I'm pretty angry about the government using our taxes to bail out Wall Street incompetence. But now I've gone from angry to pissed-off. Here's why ...

The Dot-com bubble crash wiped out $5 trillion in market value of technology companies from March 2000 to October 2002.[11]

Recent research suggests, however, that as many as 50% of the dot-coms survived through 2004, reflecting two facts: the destruction of public market wealth did not necessarily correspond to firm closings, and second, that most of the dot-coms were small players who were able to weather the financial markets storm.[12]

--http://en.wikipedia.org/wiki/Dot-com_bubble#Aftermath

Not only do I work for a dot-com survivor, I work for one of the most bruised, battered, and beaten-up dot-com survivors. I mean, just check out our stock chart! I haven't asked around the office for official corporate history or anything, but I'm pretty sure no-one remembers getting a couple billion dollars from the government while the company lost close to 99% of its market value, but I have been told there were weekly, if not daily, downsizing announcements.

Now I'm not jealous or vindictive on the part of my company - I wasn't even around for the toughest times. I'm just making a point here that market "catastrophes" and "crises" have happened, happen, and will always happen. Sadly, people will lose their jobs, their wealth, and their houses. In the case of the dot-com fallout, $5 trillion worth on Wall Street alone plus whatever subsequent losses are tallied.

6 weeks ago, the Congressional Budget Office said taxpayers would need to spend $25 billion to bail out Fannie Mae and Freddie Mac. Now we are seeing figures up to $700 billion. The figure could very well balloon to beyond the $5 trillion lost in the dot-com burst, no-one really knows - not you, not me, not Wall Street, and not Washington.

So why are we even considering footing the bill for this when no-one even knows what the total is, how it's going to be paid, or to whom we pay it? Obviously there are some very powerful financial market players gaming the system. They are not the "small players who [are] able to weather the financial markets storm."

It would be interesting to see if the Financial/Credit/Mortgage-Lending industry follows a Long Tail distribution. Might be a case of the tall head exploiting the public to avoid their necessary chop down to obscurity ...

unit tests and just-got-it-working inertia

I've been reading and enjoying The Productive Programmer by Neal Ford. It has re-ignited some of my passion for Test-Driven Development.

This morning I finished a first phase of "refactoring" some code architecture and found myself extremely hesitant to dive straight into the next phase. I think it's because the extent of my "testing" was to tab over to the fully-functioning web page and refresh after each code change. That's pretty much an "all-or-nothing" scenario.

And the thing about all-or-nothing scenarios is that once you've achieved the "all" state, you're very hesitant to go back to the "nothing" state. Maybe I'm starting to understand one of the benefits of unit tests as opposed to whole-sale acceptance tests. With smaller unit tests, you can move more concretely from nothing to something, then from something to something a little more, then finally to all done.

Brasil

One of the great things about working for a big-name web company is that you get big opportunities. I'll be speaking at PHP Conference Brasil '08 about how we use PHP at SourceForge.net. Needless to say, I'm very excited and planning some vacation time around the conference.

Open source is big in Brazil. And even more pertinent, Brazil is our 3rd-highest nation in terms of site traffic - after U.S.A. and Germany.

Hopefully I can try to overcome the language gap and present some informative material for everyone. One of the things that struck me when I joined SourceForge.net was that the site code isn't super-magic - it's really quite ordinary PHP, it's just very highly used.