Monocle Globe Society

In which the author performs technical analysis of current trends, technology, history, philosophy, politics, cooking, origami, 1st century copper smelting, great books, software and music whilst smoking a pipe and wearing an ascot.

Friday, November 06, 2009

Firefox Bookmarks & History Query API (re)Design

With the Firefox UI/UX team starting to crank out design ideas for "Places" ( a Mozilla internal name for bookmarks and history) in Firefox 3.7 and 4.0, it's high time the Places team revamped the query API.

Alex Faaborg has posted some initial UI concepts here: http://blog.mozilla.com/faaborg/2009/10/13/browsing-your-personal-web/

I have started to think about how to make an elegant API to do the heavy lifting of querying the Places database for bookmarks, history and related hierarchies. The current Places query API is not simple to use, and we want this to be simple and easily extensible by extension authors, as well as a drop in api for Jetpack.

The bug for this work is here: https://bugzilla.mozilla.org/show_bug.cgi?id=522572

One of our non-goals is to make a snap in replacement for the current API. We get to focus on the new features, like "browsing" your bookmarks and history in content-space, as well as accessing bookmarks and history via the "awesomebar".

I have posted the beginning stages of this work to the wiki, here, the Firefox "project page" is here. We are in a stage of thinking about and sketching what this simple, elegant API might look like, and we would love to get feedback and ideas from our colleagues and the Mozilla community. The Places 3.7 meta bug is here: https://bugzilla.mozilla.org/show_bug.cgi?id=523519

Friday, August 28, 2009

FunctionTimer/Timeline update

The timeline work I mentioned earlier has been going well. The patches are pretty well on their way. Until this lands, I am keeping the instrumentation patch unbitrotted each week.

On the "startup" team, we use the bugzilla whiteboard to mark all startup-related bugs with "[ts]". I wanted to propose that in the meantime before the startup patch lands, feel free to mark [ft] in the whiteboard and list any function names you want to be timed with cold/warm start, platform, etc.

I will run the [ft] query often and add the timers to the instrumentation patch(es), and update the bug with the details.

Let me know if you have any ideas or questions.

Friday, August 14, 2009

Looking for dead code

Using JSHydra to find unused functions in the Mozilla tree

JSHydra is a static analysis tool for Javascript written by Joshua Cranmer. I heard about it at one of Mozilla's Monday foundation meetings, and it sounded like it might help me introspect JS similar to the way Python's inspect module - which is a dynamic analysis tool.

Dietrich filed a bug ( 506128 ) about using JSHydra to find dead code in Firefox, especially in the startup code paths. I needed to shift gears at work, so he suggested I start using JSHydra.

There are a few tasks in order to find code via the static analysis route.

1. Build JSHydra

2. Build the preprocessed JS, XBL and XUL

(The XUL and XBL are stripped of markup and all javascripts left in the file as well as all event handlers which are defined as functions)

3. Generate the Javascript "context" by combining all Javascript files, XBL and XUL Javascript snippets for any Firefox chrome url or other type of context, Javascript modules for instance.

JSHydra operates on one file at a time, this is why all of this is needed.

4. Write Javascripts that use the JSHydra resultant AST and API to find all functions defined in a context, as well as all of the executed functions. The difference of these two lists will give us an idea of what functions may be "dead code".

Let's do this thing:

The first step is the get JSHydra built, which is built on top of Mozilla's Javascript engine. This is a relatively easy task:

1. checkout my clone of the original JSHydra repo (http://hg.mozilla.org/users/Pidgeot18_gmail.com/jshydra):

hg clone http://bitbucket.org/daviddahl/jshydra-ddahl/

2. configure the build with 2 paths: --moz-src=path/to/mozilla/checkout and --moz-obj=/path/to/mozilla/object/dir

* The above configure options will use your existing Javascript build, otherwise, the makefile will clone mozilla's JS source and take a bit longer to build.

*NOTE: I think there might be something wrong with the configure script, I had to manually add these lines to my config.mk:


MOZ_OBJDIR := /home/ddahl/code/moz/mozilla-central/obj-i686-pc-linux-gnu-optimize
MOZ_SRCDIR := /home/ddahl/code/moz/mozilla-central/mozilla


3. Run make.

This will build JShydra. To use JSHydra against a Mozilla-based project such as Firefox, you need to have all of the preprocessed Javascript built.

4. Edit jshydra-ddahl/myrules.mk:

Make sure JSTOOLDIR in myrules.mk points to /path/to/src/mozilla/config

copy jshydra-ddahl/build/ChromeHacker.py,
jshydra-ddahl/build/NicePreprocessor.py, and
jshydra-ddahl/build/myrules.mk
into /path/to/object/dir/mozilla/config

and also copy them into /path/to/source/mozilla/config

AND, copy myrules.mk into path/to/src/mozilla/js/src/config

(the js system uses a separate build system)

5. Do a full build of Firefox, and 'make jsexport'

You need to run "make jsexport" on your object dir once Firefox is built.

This step should produce a directory in /path/to/obj/dir/dist/jsfiles

The jsfiles directory will have all of the "preprocessed js, XUL and xml (XBL) files.

6. context.py and jscontext.py

I have put together some python scripts that will attempt to build a Javascript file with all of the Javascript that is in a single JS context, i.e: chrome://browser/browser.xul - or - nsBrowserGlue.js

context.py will generate a context for a chrome URL.

jscontext.py will generate a context for a JS component or module.

The reason for this is that JSHydra only operates on a single file, so context and jscontext will try to package all of the JS into one file.


python context.py --help
python jscontext.py --help for operating instructions.


7. Finding dead code:

I have written a script using the JSHydra api to try and find functions that are not called but defined in a context, this will give us some idea of where to begin looking for code that is never executed, code that might need to be removed from the tree. It is a bit inaccurate, as I have not accounted for globally available functions like setTimeout, escape, unescape, etc. and functions that are part of xpconnect components. There are also features of JSHydra that are not complete that will make this more accurate.

The dead code work in progress is in scripts/deadCode.js, run it like this:

jshydra scripts/deadCode.js /path/to/newly/built/jsContext/file.js

And you get a lot of output as it builds up lists of functions both defined and called. At the end is a summary:


####################################################################
################ JSHydra *Possible* Dead Code Summary Report #######
################ JSContext: post-build/sample_browser_xul_context.js
1073 Defined Functions
503 Functions Called
105 Functions Called But NOT Defined in this context*
423 Functions named 'undefined' - (anonymous)


It has been pretty interesting doing JS static analysis, I hope more hackers will find this worthy of working on. You can find some helpful folks on irc in #static and #startup.

Special thanks to Joshua Cranmer, David Humphrey, Taras Glek and Dietrich Ayala

Tuesday, August 04, 2009

Timeline Hacking

I have been meaning to put up a post about our Firefox startup time work. Vlad and Dietrich have been leading some efforts in trying to identify and understand any bottlenecks that occur when Firefox starts up. This is not an easy thing to do, it seems we are need of more profiling tools.

Vlad has been hacking on a set of C++ and JS function timers for little while, and I also pitched in. The patch Vlad started was both a set of timers and a lot of instrumentation. He had me split it up, clean up the timers and change how it logs the timeline data.

The Timeline data looks like this:


> XRE_main
- 190 ms ( 190 ms total) - XRE_main [ScopedXPCOMStartup]
> NS_CreateServicesFromCategory: xpcom-autoregistration (start)
<> NS_CreateServicesFromCategory: prefservice:after-app-defaults (prefservice:after-app-defaults)
<> nsIOService::Init
> nsSocketTransportService::Init
- 0 ms ( 0 ms total) - nsSocketTransportService::Init [Created thread]
- 0 ms ( 0 ms total) - nsSocketTransportService::Init [UpdatePrefs]
<> NS_CreateServicesFromCategory: xpcom-autoregistration (end)
<> NS_CreateServicesFromCategory: xpcom-startup (xpcom-startup)
<> nsChromeRegistry::Init
- 0 ms ( 0 ms total) - nsChromeRegistry::Init [NS_RegisterStaticAtoms]
- 0 ms ( 0 ms total) - nsChromeRegistry::Init [GetProtocolHandler 'jar']
- 0 ms ( 0 ms total) - nsChromeRegistry::Init [A]
- 0 ms ( 0 ms total) - nsChromeRegistry::Init [B]
> nsChromeRegistry::CheckForNewChrome
- 4 ms ( 4 ms total) - nsChromeRegistry::CheckForNewChrome [processed chrome manifest /home/ddahl/code/moz/mozilla-central/obj-i686-pc-linux-gnu-optimize/dist/bin/chrome/browser.jar]
- 4 ms ( 8 ms total) - nsChromeRegistry::CheckForNewChrome [processed chrome manifest /home/ddahl/code/moz/mozilla-central/obj-i686-pc-linux-gnu-optimize/dist/bin/chrome/browser.jar]
- 0 ms ( 8 ms total) - nsChromeRegistry::CheckForNewChrome [processed all chrome manifests]
- 0 ms ( 8 ms total) - nsChromeRegistry::CheckForNewChrome [processed all skin manifests]
<> SetWindowCreator
- 0 ms ( 0 ms total) - SetWindowCreator [RegisterFactory done]
- 0 ms ( 0 ms total) - SetWindowCreator [Got ToolkitChromeRegistry service]
- 17 ms ( 17 ms total) - SetWindowCreator [OS Accessibility check]
...

There are still a few more iterations to go before this patch is checked into Mozilla Central, but I thought I would point it out to developers since it is quite generic and can time any functions in the Mozilla source, not just startup. While vlad, brenden, wan-teh and bsmedberg are iterating on review and "the hard stuff", I will keep working on placing timers, hopefully with help from other knowledgeable Mozilla hackers.

Here is the latest wiki page on usage.

I am also working some visualization scripts and tools to make it quick to identify the bottlenecks. Right now, I am using MIT's Simile Timeline. Pretty cool stuff.

I am hoping that some seasoned Moz Hackers might know of more places to insert timers, or perhaps contribute additional targeted timing instrumentation. The cool thing, of course, is that we added an ac_add_option to turn all of this on, otherwise it is noop code, making it pretty clean to use.

If you have any questions we are all hanging out in #startup. Also, send me an email: ddahl mozilla


Monday, June 29, 2009

Fear, uncertainty and doubt

Ok, I admit it, I spent the weekend running Windows 7 on my Thinkpad. It was fun. It was exciting. Oh yeah, I was gaming...

But, the experience of installing Windows 7 on my machine was fast and easy. The experience of running Windows 7 was not Horrible. A video driver was even installed automagickally. Windows 7 seems to get out of your way, and is pretty snappy.

I'm shocked.

I am sure you are too. The bummer here is that Microsoft has actually listened to the users' wailing and gnashing of teeth over Vista. I fear for Linux adoption in the face of this. However, I am happy for Windows users - they have an OS to look forward to. ( And the price that it comes with - I hear Win 7 will not be cheap ).

Heavens! Has ddahl gone mad? It sure sounds like it.

Simmer down now! There is no replacement for Unix/Linux anywhere in the Windows world, so my dev environment will continue to be on Ubuntu, which I love using. I just wish there were speedy video drivers for my Thinkpad T500. Windows 7 is so much better (in that regard).

How are the Linux distros going to finally make headway on the desktop? I think when Vista came out there was a real opportunity to finally make a big splash, and they have, but now I have a real doubt in my mind about adoption rates. It's things like my video drivers that will drive people like me back to the warm embrace of Apple's "not as evil" business practices and non-nerds back to Microsoft.

Happy Monday!

Saturday, May 02, 2009

Counting Lines

Out of curiosity, I decided to see how many lines of code are in the Mozilla central repo.

I used CLOC, which stands for "Counting Lines of Code" a fast perl program http://cloc.sourceforge.net

ddahl-t500 ~ % perl ~/bin/cloc.pl --exclude-dir=.hg ~/code/moz/mozilla-central/mozilla
34572 text files.
33534 unique files.
56892 files ignored.

http://cloc.sourceforge.net v 1.08 T=134.0 s (204.0 files/s, 45458.4 lines/s)
--------------------------------------------------------------------------------
Language files blank comment code scale 3rd gen. equiv
--------------------------------------------------------------------------------
C++ 3096 246458 281710 1224917 x 1.51 = 1849624.67
C 1670 166078 241676 894512 x 0.77 = 688774.24
HTML 10113 92348 19579 625324 x 1.90 = 1188115.60
Javascript 4756 156677 221426 471609 x 1.48 = 697981.32
C/C++ Header 4109 100775 310865 413259 x 1.00 = 413259.00
IDL 1250 12107 0 120206 x 3.80 = 456782.80
Bourne Shell 220 16398 20518 106269 x 3.81 = 404884.89
Assembly 100 4545 2967 53678 x 0.25 = 13419.50
XML 650 14244 8639 50284 x 1.90 = 95539.60
Perl 242 6613 11643 30146 x 4.00 = 120584.00
CSS 463 7264 12306 30108 x 1.00 = 30108.00
Python 128 4880 12547 20428 x 4.20 = 85797.60
m4 22 1873 287 15602 x 1.00 = 15602.00
Java 129 2782 5736 11613 x 1.36 = 15793.68
DTD 141 2321 3028 10117 x 1.90 = 19222.30
Teamcenter def 43 60 2 4930 x 1.00 = 4930.00
SKILL 5 79 3 2680 x 2.00 = 5360.00
make 94 1554 4818 2620 x 2.50 = 6550.00
Objective C 3 235 211 959 x 2.96 = 2838.64
DOS Batch 42 175 121 621 x 0.63 = 391.23
Bourne Again Shell 5 138 452 503 x 3.81 = 1916.43
C# 4 57 261 453 x 1.36 = 616.08
Korn Shell 4 44 249 303 x 3.81 = 1154.43
Pascal 5 56 29 295 x 0.88 = 259.60
MATLAB 2 48 0 277 x 4.00 = 1108.00
Lisp 1 30 32 256 x 1.25 = 320.00
PHP 2 75 126 248 x 3.50 = 868.00
MSBuild scripts 2 2 0 242 x 1.90 = 459.80
XSLT 8 52 36 234 x 1.90 = 444.60
lex 2 64 59 223 x 1.00 = 223.00
Visual Basic 7 30 349 143 x 2.76 = 394.68
yacc 1 15 42 79 x 1.51 = 119.29
awk 3 15 46 70 x 3.81 = 266.70
SQL 1 2 0 56 x 2.29 = 128.24
sed 4 1 0 52 x 4.00 = 208.00
Ada 1 5 0 49 x 0.52 = 25.48
C Shell 2 22 13 40 x 3.81 = 152.40
D 2 7 98 16 x 1.70 = 27.20
Expect 1 0 0 1 x 2.00 = 2.00
--------------------------------------------------------------------------------
SUM: 27333 838129 1159874 4093422 x 1.50 = 6124253.00
--------------------------------------------------------------------------------


So: 4 million lines of code, wow.

What about Chrome?


ddahl-t500 ~ % perl ~/bin/cloc.pl --exclude-dir=.hg ~/code/cpp/home/chrome-svn/tarball/chromium/src
36331 text files.
34106 unique files.
118852 files ignored.

http://cloc.sourceforge.net v 1.08 T=169.0 s (100.0 files/s, 28630.1 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code scale 3rd gen. equiv
-------------------------------------------------------------------------------
C++ 4567 219341 217686 1074656 x 1.51 = 1622730.56
C 657 76387 135868 526696 x 0.77 = 405555.92
C/C++ Header 5501 142924 330982 489098 x 1.00 = 489098.00
Perl 1463 98760 135606 270429 x 4.00 = 1081716.00
Python 1202 45766 70503 200647 x 4.20 = 842717.40
Javascript 1678 37060 71865 184861 x 1.48 = 273594.28
Bourne Shell 113 18492 22420 137002 x 3.81 = 521977.62
Assembly 4 60 44 131490 x 0.25 = 32872.50
HTML 741 4491 1244 44920 x 1.90 = 85348.00
m4 18 4361 228 37068 x 1.00 = 37068.00
Objective C 185 6036 7799 29487 x 2.96 = 87281.52
IDL 342 1666 0 13779 x 3.80 = 52360.20
XML 208 604 712 6651 x 1.90 = 12636.90
CSS 30 1319 437 6492 x 1.00 = 6492.00
make 55 852 960 5860 x 2.50 = 14650.00
Tcl/Tk 23 810 1196 5507 x 4.00 = 22028.00
yacc 4 572 171 4532 x 1.51 = 6843.32
Expect 15 4 2 2113 x 2.00 = 4226.00
MATLAB 18 222 0 1937 x 4.00 = 7748.00
XSD 3 143 1129 1587 x 1.90 = 3015.30
C# 11 182 632 1226 x 1.36 = 1667.36
DOS Batch 31 138 86 649 x 0.63 = 408.87
Teamcenter def 16 29 88 521 x 1.00 = 521.00
XSLT 2 10 28 294 x 1.90 = 558.60
Korn Shell 1 39 46 223 x 3.81 = 849.63
awk 6 6 84 211 x 3.81 = 803.91
Java 4 27 0 106 x 1.36 = 144.16
MSBuild scripts 1 0 7 100 x 1.90 = 190.00
YAML 1 0 0 84 x 0.90 = 75.60
MUMPS 1 2 0 29 x 4.21 = 122.09
sed 2 0 10 26 x 4.00 = 104.00
Ruby 1 10 7 15 x 4.20 = 63.00
D 1 3 24 13 x 1.70 = 22.10
PHP 1 0 0 3 x 3.50 = 10.50
-------------------------------------------------------------------------------
SUM: 16906 660316 999864 3178312 x 1.77 = 5615500.34
-------------------------------------------------------------------------------


3.1 million lines...
Looks like they are catching up on us. By one metric anyway:P

If this doesn't seem correct let me know.

Friday, April 10, 2009

antisocial networking

The next wave in web technology should be the inverse of social networking. I wouldn't want to "brand" it 'antisocial networking', since that connotation is a bit negative, but it has a certain ring to it:)

I am amazed by the way that social networking has set aside so many people's sense of privacy. I was quite hesitant to use Facebook, but started "using" anyway in 2007, (2007?, yeah - late to the game, yadda yadda).

So there I was finally using Facebook, reconnecting with people, wasting a LOT of time. Very cool. Right? Maybe not. This question seemed to enter my mind a lot: who owns this data and what is being done with it? Obviously, it is being datamined and sold and kept forever.

I'd rather my correspondence with my friends and family not be sliced and diced and sold - and kept as a public (or private) record. Forever. Internet users should stop and think hard about how all of this technology impacts us, and how for profit companies are selling and searching and slicing and dicing our thoughts, plans, pictures, ideas, and opinions.

I want to opt out. But, I also want to communicate in a modern, high-tech, fun way.

I use Gmail, and I have to say that it is so easy to use - they even host my personal domain mail for me. I ran my own server for about 5 years, it was not fun. Spammers ruined it for me, I couldn't afford the bandwidth for the spam. Gmail to the rescue. But, again, my email is datamined, ads are shown, the data is kept forever by a for-profit company.

I want to opt out.

This is the challenge for real "social entrepreneurs": we need modern, high tech, fun communications channels like Facebook, Twitter, Gmail, GChat, but these tools should be built on top of anonymity, security and privacy, (and be open source).

This is not easy. These "privy-networking" systems have to allow anonymity, security, privacy, and establish that the user owns, can copy, move or destroy the data on a whim. Oh, and they have to be easy to use too.

The fight on the privacy front is not going so well. Researchers have even figured out how to turn anonymous data into names, addreses, and phone numbers: http://www.schneier.com/blog/archives/2009/04/identifying_peo.html

And don't get me started on "Warrentless Wiretapping", which appears to continue with gusto under our new president: http://www.eff.org/press/archives/2009/04/05

The last time I checked, you were entitled to a private conversation.

Monday, March 23, 2009

Places database generator + stats collection

At Mozilla, we need to understand how Firefox is used in the wild. Knowing what "typical" profiles are like and having automated tests that attempt to model real world situations is a big plus for writing well performing code.

Just in case anyone else needs to collect data about Firefox use or model "typical" user data for performance testing, here is how Drew and I quickly put together our "Places" toolkit.

The Sprint info page is here: https://wiki.mozilla.org/Firefox/Sprints/Places_DB_Creation_Scripts

We needed:

1. a client side script that collects places.sqlite metrics

The client side script is a Javascript written by Drew.

His script runs a bunch of aggregate SQL queries against your Places SQLite database and posts this to the collection url: https://places-stats.mozilla.com/stats

and

2. A server side script to generate a places.sqlite database based on the metrics we are collecting.

I focused on the database generation.

For now, we are doing this so we can create a test (mock) sqlite database with as many records as we wish, or based on the min, max or average of the users that post to the places-stats collection url.

So the basic flow is:

1. have users visit https://places-stats.mozilla.com and run the collection script.
2. get a large number of users (and varied types of users) posting their stats to the collection url
3. be able to produce a "power user", "average user", and "light user" places.sqlite database on the fly from data hosted at places-stats.mozilla.com

I wrote a Python script for the aggregate data collection and database generation.

To make this an easy, fast exercise in software re-use, I used Django's db module to reverse engineer the Places schema into a set of Python models.

Once you have Django set up you can run the famous 'manage.py inspectdb', which queries your SQLite db schema and outputs the corresponding django.db Python classes.

It's trivial to inject new rows into the database using django.db:


place = MozPlaces(
url=my_url,
title=my_title,
rev_host=reverse_host(my_url),
visit_count=1,
hidden=0,
typed=1,
favicon=new_favicon(),
frecency=1)
place.save()


('MozPlaces' is a django.db ORM class)

Wow, that was easy, but wait, there is more to do.

We are not even attempting to create 'real' generated place data, we just want the rows in the database to seem real. We can generate random host, domain, and tld data like this:


def url_parts():
"""
return a dictionary like: {'proto':'http'
'host':'www',
'domain':'foo',
'tld':'com'}
"""
protocol = ['https','http','ftp']
host_len = random.randint(4,26)
host = "".join(random.sample(ALPHA,host_len))
domain_len = random.randint(2,26)
domain = "".join(random.sample(ALPHA,domain_len))
tld_len = random.randint(2,3)
tld = "".join(random.sample(ALPHA,tld_len))
proto_idx = random.randint(0, 2)
proto = protocol[proto_idx]
return {'proto':proto,'host':host,'domain':domain,'tld':tld}


Python's random module has a ton of cool features. Output from the program shows that we end up with crazy looking hosts:


% python builddb/generate.py

h = httplib2.Http(os.tmpnam())
########################################################
Creating 131901 Places
Creating about 191594 History Visits
Creating about 12779 Bookmarks
Creating 101 Keywords
Creating 2173 Input History Records
########################################################
131901
Place #1 created
https://rmxwunibhvqzgjfclasypedko.zjrlundpaocs.kc/00000120269538042dedec07007f000000010001
Place #2 created
http://hlbgtm.wjxbdquyraotliek.au/000001202695391f62a5444e007f000000010001
Place #3 created
http://zdlxfpavecirty.urjawdvzoxgqemcikl.fp/00000120269539d794891209007f000000010001
Place #4 created
http://viwzykb.ofwxjmvltr.oa/0000012026953ab4b233317e007f000000010001
Place #5 created
https://yphswltjfmrbqogcd.qvd.ozd/0000012026953b539bc78b95007f000000010001
Place #6 created
ftp://pncqvksgazieuhdlofwxrtbymj.oekt.rbk/0000012026953c1f28a069ce007f000000010001
Place #7 created
http://lsmqeaojpxibvgnukwztcryhfd.isryhudzoeqjxtcankfgm.sg/0000012026953ca74487966d007f000000010001


My favorite site of the lot is "yphswltjfmrbqogcd.qvd.ozd":)

The generation script populates "Places", History, Bookmarks, Favicons, Input History and Keywords. I still have a few more entity types to generate, but this is sufficient for the testing we need to do now.

The current patch is here: https://bug480340.bugzilla.mozilla.org/attachment.cgi?id=367263

The bug is here: https://bugzilla.mozilla.org/show_bug.cgi?id=480340


The basic lesson learned is that you can build an effective, one-off data collection/metrics tool quickly and easily. I am sure others at Mozilla need tools like this, so do not hesitate to ping me with questions.

Tuesday, March 10, 2009

Orms in Firefox: yes please!

There has been a lot of discussion about ORMs, web frameworks and MozStorage on Mozilla newsgroups as of late. Coincidentally, I have been slogging through using MozStorage with Places (bookmarking) code in my day to day. I really miss my days of lazy lazy Orm-y development, you know, Django Models:


my_old_macs = Computer.objects.filter(model__exact='Mac IIci').order_by('-date_aquired')


the result object 'my_old_macs' is a wrapped query that has not executed yet. Once you begin iterating, it executes and returns the rows as Computer objects.


for mac in my_old_macs:
print mac.model
print mac.nickname
print mac.date_aquired

Ahhh, the beauty and simplicity. Here is the Model reference.

I need this kind of easy to use (and yet sophisticated) ORM style database connectivity in Firefox for the 60% + of the time where a simple, bloated ORM does the trick.

There is a related bug in Bugzilla.

I have spent a lot of time lately (mostly weekends) hacking some very buggy and naive ORM code that mimics Django - a little:)

I have attached it to bug 394372

I would love some feedback, I know I am doing some things wrong and bad, but I think I have some good concepts fleshed out.

Here is the basic usage:


var id = new Field('id','INTEGER',null,false,true,true,null);
var make_model = new Field('make_model','VARCHAR',128,false,false,false,null);
var fields = [id,make_model];
var computer = new Model(fields,'myDbTable');
var models = [computer];
var orm = new Orm('computers.sqlite',models);

// create the database:
orm.createDB();

// let's insert:
computer.save({make_model:'Mac IIci'});
computer.save({make_model:'Mac IIcx'});
computer.save({make_model:'Mac IIvx'});

// get a computer
var myIIci = computer.filter(['make_model__eq__Mac IIci']);

// Not working yet, but the style I am going for:

// update a computer
myIIci.save({make_model:'Mac IIci MK2'});

// delete a computer
myIIci.delete();

// JOIN query:

var nerdsWithIIcis = nerd.filter(['computer__make_model__eq__Mac IIci']);




Let me know what you think. You can do prety amazing things with Django, and yes, you do have to still write SQL here and there for perfomance reasons.

I hang out in #places, nick: ddahl

Cheers!

Tuesday, February 17, 2009

MozShell - another javascript shell. Cripes!

I basically re-implemented Javascript Shell for my own fun and excitement. I re-used the History functions and the enumerateWindows functions from Javascript Shell 1.4 and the Extension Developer's Extension.

I could not think of another name, so MozShell it is. Of course, I should have done a little bit more research before posting it to Google Code: http://code.google.com/p/moz-shell/

I have spent a bit of my spare time coding this up, and it is not an easy problem to solve, so pitch in and help - or, at least report bugs and ideas:)

Saturday, February 07, 2009

Tinkering with Javascript Shell

I am having a wild time trying to follow the voluminous amount of code involved in the Places component of Firefox. I finally remembered to install the Extension Developer's extension, which would not install by default as it has no "secure update method" (or some such complaint by Add-ons). Anyway, I added a couple of configuration option to about:config...

note: Javascript Shell is included in "Extension Developer's Extension".

extensions.checkCompatibility (bool) = false <-- because I am hacking on Trunk

extensions.checkUpdateSecurity (bool) = false <-- because Ext. Devel. Ext. has no secure update?

Not sure exactly, too lazy to find out:) Coming from the Python school, I always prototype and inspect code and live objects in iPython or the Python interpreter. This is a great way to become familiar with new code even before you try to read or run the test suite - especially complex code like Firefox Places. What I wouldn't do for a python "inspect"-like module for Javascript inside chrome. So anyway - I get all of this running and am in high spirits, but I noticed that the Javascript shell is not formatting anything that is spit out into it's ouput div. Tab completion on objects shows you a long sinlge line of each completable item, making this basically unusable (for me). I also started playing with Xush, but, without tab completion, a lightweight stand-alone window and a few bugs on Linux (which I think have been rectified), I figured I would add some css and js tweaks to make JS shell more my style. Here is what I tweaked:

1. Made all fonts 1em and 'monospace' font-family. Like a shell should be:)

2. Check the output to see if it is a function, if so, display in a pre element.

3. Added "prettyprint" source code beautifier so the above-mentioned functions are easier on the eyes - not "emacs classic theme" or anything, but a step in the "iPython" direction.

4. Added ctrl-a and ctrl-e key commands to the input widget. Yay!

5. Open the shell in a 800 x 600 window

6. Added a promt like the xpcshell promt: js>

This makes jsshell just a tad bit easier to use and cleaner to boot. I like that it is so lightweight - I tried using ChromeBug too, and it is getting faster, but it can be a bit flaky.

I just think nothing can beat a very lightweight, responsive shell.

I am sure I will keep tweaking it. If you want to get a copy, I have posted my extensiondev.jar that is part of the Extension deveoper's extension here: extensiondev.jar

postscript:

If I had bothered to look (i'm slow like that) at the google code page for Extension Developer's Extension, ( http://code.google.com/p/extensiondev/ ) I would have seen that the formatting bug was reported and the submitted patch was a simple css fix!

Well, it was fun. I may have to join the project:)

Thursday, January 22, 2009

Shut up and code

A really huge frustration I have had at work in the past is when a company or team I work with won't shut up about ideas and just code something.

I have worked on teams where people just go around and around on why something is a dumb idea or why it won't work 75% of the time or whatever.

I have found that a live demo speaks volumes about your concept and you better just stop talking and create it - even if it turns out to be unusable in the long run.

Do it and do it fast. If you are successful, great, if not, trash it and keep going.

This post by Paul Buchheit is perfect:

http://paulbuchheit.blogspot.com/2009/01/communicating-with-code.html

Saturday, January 17, 2009

irc as a way of life

It's quite amazing how much traction we get out of irc at work. It's pretty much a rule to keep the conversation inside irc. This makes a whole lot of sense, as we have all kinds of documentation in the chat logs. I am very accustomed to chatting for work conversation, debugging and whatnot. I love the decentralized model. This is above and beyond what I have ever experienced. So cool.

Between irc, blogs and bugzilla, we are all having a lot of online conversations. Email is important, but secondary. I likes. 99% of the communication is captured in a pretty meaningful way, unlike typical corporate massive reliance on email, which is so full of spam and nonsense.

Wednesday, January 07, 2009

Mozilla workflow

The cool thing about really getting your hands dirty deep inside of chrome is learning how the Mozilla developers workflow works. So cool. As an extension developer, I was so used to a workflow where I code, restart Firefox and test. When you work on Mozilla you code/write tests, run make, make check on the module or modules you are touching and start your build, then test if need be...

So cool. I am having a ton of fun. At my last job I was leading a project and now I am back in the position as a student, my favorite place to be:)

Monday, January 05, 2009

First Day @ Mozilla

Arrived 8:30. Benefits, meet new boss again, collect new ThinkPad! w00t!

Install Ubuntu 8.10, all goes well (bad cd media withstanding). apt-get installed all Mozilla dev dependencies.

Meeting.

Lunch.

Meeting.

hg clone mozilla source. starting to play with building Firefox. I am going to learn a lot.

This is exciting. I am hyped

Saturday, January 03, 2009

The Beeb is OK now and again


Check out this BBC documentary about Islamic medieval Europe. So cool. I visited Alhambra in Granada back in 2005, I was blown away.



2009 - Time to begin

I figure I have been sitting on the "blogging sidelines" long enough. I am also starting a new job that has very little secrecy involved *for once*. Yay!

I am going to try and post technical related things that will help me later and I hope others as well. I plan on getting sidetracked by many other topics as well.

Regards,

David

Wednesday, May 03, 2006

eFax sucks

I wanted to cancel my efax account... it is very difficult to do. You have to chat with some offshore worker online and have to threaten them with reversed creditcard charges and BBB complaints:

Begin Painful Transcript:

Welcome to chat.
The session has been accepted.
{Craig H.} Hello, David. Welcome to our j2 Global online chat service. I am Craig, your Online Support Representative.
{Craig H.} I'm sorry to hear that you wish to cancel the account. Could you please provide me your Fax number and PIN for verification purposes?
{david} # is (312) 275-****
{david} looking up pin now
{Craig H.} Okay.
{david} pin is ****
{Craig H.} Thank you for the information. Please give me a moment while I pull up your account. In the meanwhile, may I ask you why are canceling the number?
{david} I never use fax anymore
{Craig H.} David, I can understand that currently you do not need the service, since you don't use much. In the current situation as a special consideration, we will waive off the monthly fee for two months. You can use the fax service without paying any monthly fees for the next two billing cycles. Please feel free to contact us at any time. This way you will be able to keep your local fax number, which will enable you to send and receive any pending faxes with your number.
{Craig H.} Your eFax account will be credited with $**.** so that you may utilize our services without being billed our monthly fee for the next two billing cycles.
{Craig H.} Since you will not be charged any monthly fees for the next two months you may keep the account till then. If at all you find that you need our services during this period, then you will still have the account. If however, you still feel that you do not have any use for our services by the end of the two months credit period, then you can always contact us back anytime.
{david} No thanks. you need to cancel it now. thanks.


*** 2 - 3 minutes go by ***


{david} are you cancelling my account?
{Craig H.} David, I understand that you don't need the service. You have been offered two months credit since your are our esteemed customer. Since you have already paid for the month, I suggest you to retain this number at least till the end of two months credit offer as you will not be charged any monthly fee for the two months.
{Craig H.} I suggest you to make use of your account till the end of credit period.
{Craig H.} As a good will gesture to continue our association we will offer you an additional gift balance of $10 along with the monthly credit which will enable to send up to 100 additional fax pages free of cost anywhere in USA & Canada. It could indeed be very helpful (as we have found so, with many customers) to keep the account during this period to see if the need ever arises for our services. If so, you will still have the account. If not, then no harm has done. You can always contact us back at the end of this period and intimate us of your decision to continue with the service or not.
{Craig H.} After completion of the 2 months credit period, if you feel that the fax number does not serve your purpose, you can get back to us immediately without any further obligation to stay back. We are available 24 hours a day and 7 days a week. Please feel free to contact us at any time.

*** 30 seconds go by ***

{david} if you do not cancel my account today i am going to ask my credit card company to place a hold on all future charges
{david} you need to cancel it now, thank you
{david} I will also report you to the better business bureau. i seems like you are trying to pull a fast one on your customers by not allowing cacellation via the "may account" login on your website
{Craig H.} Daid, I am sorry to make you upset. I sincerely apologize for any inconvenience caused to you.
{Craig H.} Just wanted you to take benefit of 2 month credit period, till you finally decide.

*** Another pause ***

{david} so you are cancelling my account now right?
{Craig H.} As per your request, your account have been cancelled. Billing for your number (312) 275-**** will cease immediately.
{david} I have finally decided that my accounjt is not needed, you should cancel my account now, thank you.
{Craig H.} I’m sorry that you are leaving eFax. At eFax, we are continuously improving our products and services. Please do consider us if your faxing needs should change in the future.
{Craig H.} Is there anything else you would like me to assist you with?
{david} absolutely not


End of painful transcript...

what assholes

Last modified: Wed May 3 11:14:54 CDT 2006

Monday, May 16, 2005

IMG_2245.JPG


IMG_2245.JPG
Originally uploaded by daviddahl.
thisis a test picture blog entry

Followers

About Me

David Dahl
San Francisco, CA
Firefox Engineer at Mozilla Corp
View my complete profile