Engineering Code

User Interface ambiguity - never good

2010-06-21T17:12:00.000-07:00

This browser dialog was produced as part of the flow of stopping gas service. The text and buttons combine for almost perfect ambiguity. Continue to stop the service? Continue the service? Cancel stopping the service or Cancel the service?

You have elected to stop your NSTAR service. Do you wish to continue?

      Cancel          OK

Much clearer would be text and button labels such as:

You have elected to stop your NSTAR service. Please confirm that choice:

   Yes - please stop my service
   No - I have changed my mind

Even a simple yes / no would be so much better.

Rubinius 1.0 at RailsConf

2010-06-09T08:39:00.000-07:00

Rubinus is currently at 1.0.1 as of last week.

No continuations but otherwise Ruby 1.8.7 compatibility.

Faster string method performance through better algorithms than MRI and the just-in-time compiler.

DataMapper 1.0 was demonstrated with Rubinius 1.0 at RailsConf.

Rubinius has a built-in profiler -Xprofile. Also a sampling profiler which will show a high-level view of where your code spends its time - but with a negligible CPU overhead.

The Debugger is targeted to be fixed for 1.1. There is a Ruby API which can be used to integrate another debugger into Rubinius' debug support.

A new Query Agent API which can provide information about the state of the VM even when it is otherwise unresponsive. To be used with a control console (bin/rbx -Xagent.start and bin/rbx console) a port file is produced which the console reads to find how to communicate with the query agent. The console has commands such as pid, mem, set, threads, name and get. The latter gets variable values e.g. get system.backtrace produces the stack trace of the thread which is waiting for user input or whatever else the VM happens to be doing. This interface is very basic but has a lot of potential with future debugger/console integration.

Real Software Engineering (at RailsConf)

2010-06-08T07:59:00.000-07:00

Glenn Vanderburg is analyzing how software engineering was defined by in the late 1960s and how we got to where we are today. If software engineering is not defined by the waterfall development process or other denigrated process then how is it defined?

"Cost is always an object." - there is a price to 'do things right' and sometimes it is too high.

"Software engineering will be different from other engineering."

Traditional engineering produces a design which a manufacturing group turns into a product. In software engineering a similar process has not been very successful (Software Architects passing off designs to Programmers for implementation).

Reeves suggests that, with the cost of prototyping so low, that testing models to verify that they meet requirements is practical (and much cheaper than the example of testing the wings of a Boeing 777 until they broke).

Vanderburg mentions that Agile practices are criticized by some for being bad engineering - cowboy coding because of the flexibility of the process.

Software engineering assumptions which are now invalid: code is hard to read, to change, and that testing is expensive.

His thesis is that Real Software Engineering is unlike creating bridge designs but retains a process which delivers a product which meets requirements where the key engineering document is the source code.

Rails 3 Deep Dive - part 2

2010-06-07T12:39:00.000-07:00

Plugins

Rails 3 uses Railties to integrate your plugin/gem more cleanly. Having your plugins be gems is a goal.

Engines

Lots of configuration possibilities. There is now much more fine-grained support for hooks into the Rails startup process. You now inherit from Rails::Engine to initiate the integration.

Jeremy adds a small sample engine very quickly. Interestingly he appears to be running on Rails 3 on ruby 1.9.2-head - a vote of confidence in the recently attained support.

Generators

Now generators have simpler syntax but all the public methods are generator tasks, so make other code private.

Thor is behind the generators and has a lot of built-in support for file system and other operations.

It is easy to configure the components which you want to be used with your custom generators:

view template - say :haml
test framework - say :rspec
fixture replacement - say :factory_girl

The generators use this configuration to create appropriate files (e.g. unit tests with the correct testing framework when a new model is generated).

Rails 3 Deep Dive

2010-06-07T11:46:00.000-07:00

Surprisingly, Sinatra was a major part of the first lab in the Rails 3 Deep Dive. The idea was to understand how to use Rack with middleware and Rails, using Sinatra as an easy introduction. I love to learn new stuff but, if you want to teach Rack this way, please give us a heads-up so that ignorance of Sinatra won't slow us down.

The takeaway: Rails 3 uses Rack much more than Rails 2. A lot of Rails filters may or have become Rack middleware.

Git Immersion

2010-06-07T10:27:00.000-07:00

The tutorial was good for people who had never used git before. It also gave a good background on the implementation strategy. However, everyone there had used git before so I think a lot of the introductory material about add, commit, status etc would have been too familiar for most.

I zipped through the lab exercises and moved to Brian's Rails 3 Ropes tutorial during the mid-morning break. Over there, I had to race to catch up but got some fun experience upgrading a Rails 2 application to Rails 3. The second half was well-presented (and I will assume the first half was too) by the Envy Labs folks with Gregg Pollack doing the speaking and the rest of the team assisting people as they had problems with the exercises.

RailsConf Baltimore

2010-06-07T03:54:00.000-07:00

I have been using Ruby on Rails since just before 1.0 and yet now is the first time I have been able to get to RailsConf. It is also my first visit to Baltimore and my first trip on Amtrak's Acela brought us down here from Boston. RBM sent Brian and myself down here and, given the very productive rail journey, we are already off to a good start.

We have tried to split a lot of the conference content between us. Monday is tutorial day and while Brian is getting an introduction to Rails 3 via Rails 3 Ropes, I am going to a Git Immersion tutorial. This afternoon there is a deep dive into Rails 3 which I am looking forward to.

We have using git for a couple of months now and, while we are very confident in how we are using it, git has far more options that Subversion and I really want to understand more of the possible ways to use it.

2010-05-15T11:19:00.000-07:00

Today I helped out at a Cambridge Rails Bugmash. Along the way I noted some changes...

Within Rails the bin directory has moved from vendor/rails/railties/bin/ to vendor/rails/bin. There are plenty of online references to the old location.

I had some problems with instructions which included references to using sudo to install gems. Since I was using rvm, I needed to not use sudo in each case.

If planning to do some bug mashing on Rails I would recommend ensuring that you had Postgresql, sqlite and mysql set up correctly in advance. Also some familiarity with gems and bundler will help.

Craig Krigsman and Dan Pickett were there too.

A Happy New Year

2009-12-31T23:53:00.000-08:00

In a departure from my normal silence - a post about New Year's Eve.

Amanda Fucking Palmer lured William and Samantha and I to Symphony Hall in Boston for her gig with the Boston Pops. Plenty of other entertainment (Sxip Shirey and Miss Tess and the Bon Ton Parade were ones I heard) before the Pops started the first half of their performance with pieces requested by AFP fans.

Imperial March, from Star Wars Episode V (Williams)
All That Jazz, from Chicago (Kander.Ebb-Besterman)
It Don't Mean a Thing (If It Ain't Got That Swing) (Ellington-Nestico)
Mack the Knife (Weill/Brecht-Mason)

Followed by a world premiere of a piece by Sxip Shirey called Melody for Lizzie. And then:

Overture from Selmasongs: Dancer in the Dark (Björk)

The world premiere of a short film by Michael Pope commisioned by the Pops for the event. And then:

Suite from Carmen (Bizet)

And then Neil Gaimen's Statuesque - a short film created for televison in the UK - having its U.S. premiere.

Last, but certainly not least was Amanda. She sang quite a few songs from her repertioire but what made it memorable was the performance. Cabaret? Burlesque in parts? Entertaining throughout. The highlight for me was the first number - 'Missed Me' - as she wound through the audience coming upon various musicians who all start hidden under large cloths and then persued her with their instruments when revealed. One of those musicians was Neil Gaiman, playing the chain saw - it was that kind of a performance - memorable.

While Samantha went off to party with Michael Pope and friends and then, later, AFP and company, William and I met Valerie, Patrick Rothfuss's assistant. You may have heard of the Fabulous Lorrain or Beth of All Trades. You need to meet The Fabulous Valerie of All Trades - sweet, charming and so much fun! Clearly just what Patrick needs to become an even more efficent writing machine!

Rake and class variables

2009-02-06T08:52:00.000-08:00

I wrote a set of Rake task recently. To allow them to form a basis for future expansion, I broke them down into parts and used a class variable to hold state, since I could not pass parameters between them.

It appears that in moving to the version of Rake with Rails 2.1 that the global variables are lost at the end of each task.

I was able to re-factor the code to use a common method for re-use. However, the re-use is limited. I can call the common method from each task. However the tasks can not depend on one another since the state cannot be persisted between them.

This change made Rake tasks less useful - to me at least.

Patching ruby 1.8.6-p287 for memory profiling

2009-01-22T07:25:00.000-08:00

We use ruby 1.8.6. The current patch level is 287.

This post about memory profiling claims that the ruby patch provided will apply to 1.8.6. It did not work for me - three of the changes to gc.c failed to apply. Manual editing was straightforward.

This command should patch gc.c:

patch -p0 < ../ruby186gc.patch

For lack of a better place to put it, here is the patch file (ruby186gc.patch):
--- gc.c 2008-08-03 23:16:55.000000000 -0400
+++ gc.c 2009-01-22 09:05:26.000000000 -0500
@@ -22,8 +22,16 @@
#include
#include

+#ifdef _WIN32
+#include
+#else
+#include
+#endif
+
#ifdef HAVE_SYS_TIME_H
#include
+#elif defined(_WIN32)
+#include
#endif

#ifdef HAVE_SYS_RESOURCE_H
@@ -40,7 +48,6 @@
#if !defined(setjmp) && defined(HAVE__SETJMP)
#define setjmp(env) _setjmp(env)
#endif
-
/* Make alloca work the best possible way. */
#ifdef __GNUC__
# ifndef atarist
@@ -76,6 +83,20 @@

NORETURN(void rb_exc_jump _((VALUE)));

+static unsigned long live_objects = 0;
+unsigned long rb_os_live_objects()
+{ return live_objects; }
+
+#if defined(HAVE_LONG_LONG)
+static unsigned long long allocated_objects = 0;
+unsigned long long rb_os_allocated_objects()
+{ return allocated_objects; }
+#else
+static unsigned long allocated_objects = 0;
+unsigned long rb_os_allocated_objects()
+{ return allocated_objects; }
+#endif
+
void
rb_memerror()
{
@@ -93,6 +114,10 @@
rb_exc_raise(nomem_error);
}

+long gc_allocated_size = 0;
+long gc_num_allocations = 0;
+static int gc_statistics = 0;
+
void *
ruby_xmalloc(size)
long size;
@@ -115,6 +140,11 @@
rb_memerror();
}
}
+ if (gc_statistics) {
+ gc_allocated_size += size;
+ gc_num_allocations += 1;
+ }
+
malloc_increase += size;

return mem;
@@ -165,8 +195,16 @@
RUBY_CRITICAL(free(x));
}

+#if HAVE_LONG_LONG
+#define GC_TIME_TYPE LONG_LONG
+#else
+#define GC_TIME_TYPE long
+#endif
+
extern int ruby_in_compile;
static int dont_gc;
+static GC_TIME_TYPE gc_time = 0;
+static int gc_collections = 0;
static int during_gc;
static int need_call_final = 0;
static st_table *finalizer_table = 0;
@@ -179,7 +217,7 @@
* Enables garbage collection, returning true if garbage
* collection was previously disabled.
*
- * GC.disable #=> false
+ * GC.disable #=> false or true
* GC.enable #=> true
* GC.enable #=> false
*
@@ -215,6 +253,140 @@
return old;
}

+/*
+ * call-seq:
+ * GC.enable_stats => true or false
+ *
+ * Enables garbage collection statistics, returning true if garbage
+ * collection statistics was already enabled.
+ *
+ * GC.enable_stats #=> false or true
+ * GC.enable_stats #=> true
+ *
+ */
+
+VALUE
+rb_gc_enable_stats()
+{
+ int old = gc_statistics;
+ gc_statistics = Qtrue;
+ return old;
+}
+
+/*
+ * call-seq:
+ * GC.disable_stats => true or false
+ *
+ * Disables garbage collection statistics, returning true if garbage
+ * collection statistics was already disabled.
+ *
+ * GC.disable_stats #=> false or true
+ * GC.disable_stats #=> true
+ *
+ */
+
+VALUE
+rb_gc_disable_stats()
+{
+ int old = gc_statistics;
+ gc_statistics = Qfalse;
+ gc_allocated_size = 0;
+ gc_num_allocations = 0;
+ return old;
+}
+
+/*
+ * call-seq:
+ * GC.clear_stats => nil
+ *
+ * Clears garbage collection statistics, returning nil. This resets the number
+ * of collections (GC.collections) and the time used (GC.time) to 0.
+ *
+ * GC.clear_stats #=> nil
+ *
+ */
+
+VALUE
+rb_gc_clear_stats()
+{
+ gc_collections = 0;
+ gc_time = 0;
+ gc_allocated_size = 0;
+ gc_num_allocations = 0;
+ return Qnil;
+}
+
+/*
+ * call-seq:
+ * GC.allocated_size => Integer
+ *
+ * Returns the size of memory (in bytes) allocated since GC statistics collection
+ * was enabled.
+ *
+ * GC.allocated_size #=> 35
+ *
+ */
+VALUE
+rb_gc_allocated_size()
+{
+ return INT2NUM(gc_allocated_size);
+}
+
+/*
+ * call-seq:
+ * GC.num_allocations => Integer
+ *
+ * Returns the number of memory allocations since GC statistics collection
+ * was enabled.
+ *
+ * GC.num_allocations #=> 150
+ *
+ */
+VALUE
+rb_gc_num_allocations()
+{
+ return INT2NUM(gc_num_allocations);
+}
+
+/*
+ * call-seq:
+ * GC.collections => Integer
+ *
+ * Returns the number of garbage collections performed while GC statistics collection
+ * was enabled.
+ *
+ * GC.collections #=> 35
+ *
+ */
+
+VALUE
+rb_gc_collections()
+{
+ return INT2NUM(gc_collections);
+}
+
+/*
+ * call-seq:
+ * GC.time => Integer
+ *
+ * Returns the time spent during garbage collection while GC statistics collection
+ * was enabled (in micro seconds).
+ *
+ * GC.time #=> 20000
+ *
+ */
+
+VALUE
+rb_gc_time()
+{
+#if HAVE_LONG_LONG
+ return LL2NUM(gc_time);
+#else
+ return LONG2NUM(gc_time);
+#endif
+}
+
+
VALUE rb_mGC;

static struct gc_list {
@@ -306,7 +478,7 @@
static RVALUE *freelist = 0;
static RVALUE *deferred_final_list = 0;

-#define HEAPS_INCREMENT 10
+static int heaps_increment = 10;
static struct heaps_slot {
void *membase;
RVALUE *slot;
@@ -315,13 +487,165 @@
static int heaps_length = 0;
static int heaps_used = 0;

-#define HEAP_MIN_SLOTS 10000
-static int heap_slots = HEAP_MIN_SLOTS;
+static int heap_min_slots = 10000;
+static int heap_slots = 10000;
+
+static int heap_free_min = 4096;
+static int heap_slots_increment = 10000;
+static double heap_slots_growth_factor = 1.8;

-#define FREE_MIN 4096
+static long initial_malloc_limit = GC_MALLOC_LIMIT;
+
+static int verbose_gc_stats = Qfalse;
+
+static FILE* gc_data_file = NULL;

static RVALUE *himem, *lomem;

+static void set_gc_parameters()
+{
+ char *gc_stats_ptr, *min_slots_ptr, *free_min_ptr, *heap_slots_incr_ptr,
+ *heap_incr_ptr, *malloc_limit_ptr, *gc_heap_file_ptr, *heap_slots_growth_factor_ptr;
+
+ gc_data_file = stderr;
+
+ gc_stats_ptr = getenv("RUBY_GC_STATS");
+ if (gc_stats_ptr != NULL) {
+ int gc_stats_i = atoi(gc_stats_ptr);
+ if (gc_stats_i > 0) {
+ verbose_gc_stats = Qtrue;
+ }
+ }
+
+ gc_heap_file_ptr = getenv("RUBY_GC_DATA_FILE");
+ if (gc_heap_file_ptr != NULL) {
+ FILE* data_file = fopen(gc_heap_file_ptr, "w");
+ if (data_file != NULL) {
+ gc_data_file = data_file;
+ }
+ else {
+ fprintf(stderr,
+ "can't open gc log file %s for writing, using default\n", gc_heap_file_ptr);
+ }
+ }
+
+ min_slots_ptr = getenv("RUBY_HEAP_MIN_SLOTS");
+ if (min_slots_ptr != NULL) {
+ int min_slots_i = atoi(min_slots_ptr);
+ if (verbose_gc_stats) {
+ fprintf(gc_data_file, "RUBY_HEAP_MIN_SLOTS=%s\n", min_slots_ptr);
+ }
+ if (min_slots_i > 0) {
+ heap_slots = min_slots_i;
+ heap_min_slots = min_slots_i;
+ }
+ }
+
+ free_min_ptr = getenv("RUBY_HEAP_FREE_MIN");
+ if (free_min_ptr != NULL) {
+ int free_min_i = atoi(free_min_ptr);
+ if (verbose_gc_stats) {
+ fprintf(gc_data_file, "RUBY_HEAP_FREE_MIN=%s\n", free_min_ptr);
+ }
+ if (free_min_i > 0) {
+ heap_free_min = free_min_i;
+ }
+ }
+
+ heap_incr_ptr = getenv("RUBY_HEAP_INCREMENT");
+ if (heap_incr_ptr != NULL) {
+ int heap_incr_i = atoi(heap_incr_ptr);
+ if (verbose_gc_stats) {
+ fprintf(gc_data_file, "RUBY_HEAP_INCREMENT=%s\n", heap_incr_ptr);
+ }
+ if (heap_incr_i > 0) {
+ heaps_increment = heap_incr_i;
+ }
+ }
+
+ heap_slots_incr_ptr = getenv("RUBY_HEAP_SLOTS_INCREMENT");
+ if (heap_slots_incr_ptr != NULL) {
+ int heap_slots_incr_i = atoi(heap_slots_incr_ptr);
+ if (verbose_gc_stats) {
+ fprintf(gc_data_file, "RUBY_HEAP_SLOTS_INCREMENT=%s\n", heap_slots_incr_ptr);
+ }
+ if (heap_slots_incr_i > 0) {
+ heap_slots_increment = heap_slots_incr_i;
+ }
+ }
+
+ heap_slots_growth_factor_ptr = getenv("RUBY_HEAP_SLOTS_GROWTH_FACTOR");
+ if (heap_slots_growth_factor_ptr != NULL) {
+ double heap_slots_growth_factor_d = atoi(heap_slots_growth_factor_ptr);
+ if (verbose_gc_stats) {
+ fprintf(gc_data_file, "RUBY_HEAP_SLOTS_GROWTH_FACTOR=%s\n", heap_slots_growth_factor_ptr);
+ }
+ if (heap_slots_growth_factor_d > 0) {
+ heap_slots_growth_factor = heap_slots_growth_factor_d;
+ }
+ }
+
+ malloc_limit_ptr = getenv("RUBY_GC_MALLOC_LIMIT");
+ if (malloc_limit_ptr != NULL) {
+ int malloc_limit_i = atol(malloc_limit_ptr);
+ if (verbose_gc_stats) {
+ fprintf(gc_data_file, "RUBY_GC_MALLOC_LIMIT=%s\n", malloc_limit_ptr);
+ }
+ if (malloc_limit_i > 0) {
+ initial_malloc_limit = malloc_limit_i;
+ }
+ }
+}
+
+/*
+ * call-seq:
+ * GC.dump => nil
+ *
+ * dumps information about the current GC data structures to the GC log file
+ *
+ * GC.dump #=> nil
+ *
+ */
+
+VALUE
+rb_gc_dump()
+{
+ int i;
+
+ for (i = 0; i < heaps_used; i++) {
+ int heap_size = heaps[i].limit;
+ fprintf(gc_data_file, "HEAP[%2d]: size=%7d\n", i, heap_size);
+ }
+
+ return Qnil;
+}
+
+/*
+ * call-seq:
+ * GC.log String => String
+ *
+ * Logs string to the GC data file and returns it.
+ *
+ * GC.log "manual GC call" #=> "manual GC call"
+ *
+ */
+
+VALUE
+rb_gc_log(self, original_str)
+ VALUE self, original_str;
+{
+ if (original_str == Qnil) {
+ fprintf(gc_data_file, "\n");
+ }
+ else {
+ VALUE str = StringValue(original_str);
+ char *p = RSTRING(str)->ptr;
+ fprintf(gc_data_file, "%s\n", p);
+ }
+ return original_str;
+}
+
+
static void
add_heap()
{
@@ -332,7 +656,7 @@
struct heaps_slot *p;
int length;

- heaps_length += HEAPS_INCREMENT;
+ heaps_length += heaps_increment;
length = heaps_length*sizeof(struct heaps_slot);
RUBY_CRITICAL(
if (heaps_used > 0) {
@@ -348,10 +672,10 @@
for (;;) {
RUBY_CRITICAL(p = (RVALUE*)malloc(sizeof(RVALUE)*(heap_slots+1)));
if (p == 0) {
- if (heap_slots == HEAP_MIN_SLOTS) {
+ if (heap_slots == heap_min_slots) {
rb_memerror();
}
- heap_slots = HEAP_MIN_SLOTS;
+ heap_slots = heap_min_slots;
continue;
}
heaps[heaps_used].membase = p;
@@ -367,8 +691,9 @@
if (lomem == 0 || lomem > p) lomem = p;
if (himem < pend) himem = pend;
heaps_used++;
- heap_slots *= 1.8;
- if (heap_slots <= 0) heap_slots = HEAP_MIN_SLOTS;
+ heap_slots += heap_slots_increment;
+ heap_slots_increment *= heap_slots_growth_factor;
+ if (heap_slots <= 0) heap_slots = heap_min_slots;

while (p < pend) {
p->as.free.flags = 0;
@@ -402,6 +727,8 @@
RANY(obj)->file = ruby_sourcefile;
RANY(obj)->line = ruby_sourceline;
#endif
+ live_objects++;
+ allocated_objects++;
return obj;
}

@@ -1030,6 +1357,39 @@
}
}

+static char* obj_type(int tp)
+{
+ switch (tp) {
+ case T_NIL : return "NIL";
+ case T_OBJECT : return "OBJECT";
+ case T_CLASS : return "CLASS";
+ case T_ICLASS : return "ICLASS";
+ case T_MODULE : return "MODULE";
+ case T_FLOAT : return "FLOAT";
+ case T_STRING : return "STRING";
+ case T_REGEXP : return "REGEXP";
+ case T_ARRAY : return "ARRAY";
+ case T_FIXNUM : return "FIXNUM";
+ case T_HASH : return "HASH";
+ case T_STRUCT : return "STRUCT";
+ case T_BIGNUM : return "BIGNUM";
+ case T_FILE : return "FILE";
+
+ case T_TRUE : return "TRUE";
+ case T_FALSE : return "FALSE";
+ case T_DATA : return "DATA";
+ case T_MATCH : return "MATCH";
+ case T_SYMBOL : return "SYMBOL";
+
+ case T_BLKTAG : return "BLKTAG";
+ case T_UNDEF : return "UNDEF";
+ case T_VARMAP : return "VARMAP";
+ case T_SCOPE : return "SCOPE";
+ case T_NODE : return "NODE";
+ default: return "____";
+ }
+}
+
static void
free_unused_heaps()
{
@@ -1059,13 +1419,23 @@
int i;
unsigned long live = 0;
unsigned long free_min = 0;
+ live_objects = 0;
+
+ unsigned long really_freed = 0;
+ int free_counts[256];
+ int live_counts[256];
+ int do_gc_stats = gc_statistics & verbose_gc_stats;

for (i = 0; i < heaps_used; i++) {
free_min += heaps[i].limit;
}
free_min = free_min * 0.2;
- if (free_min < FREE_MIN)
- free_min = FREE_MIN;
+ if (free_min < heap_free_min)
+ free_min = heap_free_min;
+
+ if (do_gc_stats) {
+ for (i = 0 ; i< 256; i++) { free_counts[i] = live_counts[i] = 0; }
+ }

if (ruby_in_compile && ruby_parser_stack_on_heap()) {
/* should not reclaim nodes during compilation
@@ -1098,6 +1468,9 @@
if (!(p->as.basic.flags & FL_MARK)) {
if (p->as.basic.flags) {
obj_free((VALUE)p);
+ if (do_gc_stats) {
+ really_freed++;
+ }
}
if (need_call_final && FL_TEST(p, FL_FINALIZE)) {
p->as.free.flags = FL_MARK; /* remain marked */
@@ -1105,6 +1478,12 @@
final_list = p;
}
else {
+ if (do_gc_stats) {
+ int obt = p->as.basic.flags & T_MASK;
+ if (obt) {
+ free_counts[obt]++;
+ }
+ }
p->as.free.flags = 0;
p->as.free.next = freelist;
freelist = p;
@@ -1117,7 +1496,10 @@
}
else {
RBASIC(p)->flags &= ~FL_MARK;
- live++;
+ live_objects++;
+ if (do_gc_stats) {
+ live_counts[RANY((VALUE)p)->as.basic.flags & T_MASK]++;
+ }
}
p++;
}
@@ -1135,8 +1517,8 @@
}
}
if (malloc_increase > malloc_limit) {
- malloc_limit += (malloc_increase - malloc_limit) * (double)live / (live + freed);
- if (malloc_limit < GC_MALLOC_LIMIT) malloc_limit = GC_MALLOC_LIMIT;
+ malloc_limit += (malloc_increase - malloc_limit) * (double)live_objects / (live_objects + freed);
+ if (malloc_limit < initial_malloc_limit) malloc_limit = initial_malloc_limit;
}
malloc_increase = 0;
if (freed < free_min) {
@@ -1144,6 +1526,20 @@
}
during_gc = 0;

+ if (do_gc_stats) {
+ fprintf(gc_data_file, "objects processed: %.7d\n", live+freed);
+ fprintf(gc_data_file, "live objects : %.7d\n", live);
+ fprintf(gc_data_file, "freelist objects : %.7d\n", freed - really_freed);
+ fprintf(gc_data_file, "freed objects : %.7d\n", really_freed);
+ for(i=0; i<256; i++) {
+ if (free_counts[i]>0) {
+ fprintf(gc_data_file,
+ "kept %.7d / freed %.7d objects of type %s\n",
+ live_counts[i], free_counts[i], obj_type(i));
+ }
+ }
+ }
+
/* clear finalization list */
if (final_list) {
deferred_final_list = final_list;
@@ -1338,6 +1734,7 @@
struct gc_list *list;
struct FRAME * volatile frame; /* gcc 2.7.2.3 -O2 bug?? */
jmp_buf save_regs_gc_mark;
+ struct timeval gctv1, gctv2;
SET_STACK_END;

#ifdef HAVE_NATIVETHREAD
@@ -1354,6 +1751,14 @@
if (during_gc) return;
during_gc++;

+ if (gc_statistics) {
+ gc_collections++;
+ gettimeofday(&gctv1, NULL);
+ if (verbose_gc_stats) {
+ fprintf(gc_data_file, "Garbage collection started\n");
+ }
+ }
+
init_mark_stack();

gc_mark((VALUE)ruby_current_node, 0);
@@ -1429,6 +1834,17 @@
} while (!MARK_STACK_EMPTY);

gc_sweep();
+
+ if (gc_statistics) {
+ GC_TIME_TYPE musecs_used;
+ gettimeofday(&gctv2, NULL);
+ musecs_used = ((GC_TIME_TYPE)(gctv2.tv_sec - gctv1.tv_sec) * 1000000) + (gctv2.tv_usec - gctv1.tv_usec);
+ gc_time += musecs_used;
+
+ if (verbose_gc_stats) {
+ fprintf(gc_data_file, "GC time: %d msec\n", musecs_used / 1000);
+ }
+ }
}

void
@@ -1610,6 +2026,7 @@
if (!rb_gc_stack_start) {
Init_stack(0);
}
+ set_gc_parameters();
add_heap();
}

@@ -2031,6 +2448,35 @@
return (VALUE)((long)obj|FIXNUM_FLAG);
}

+/* call-seq:
+ * ObjectSpace.live_objects => number
+ *
+ * Returns the count of objects currently allocated in the system. This goes
+ * down after the garbage collector runs.
+ */
+static
+VALUE os_live_objects(VALUE self)
+{ return ULONG2NUM(live_objects); }
+
+/* call-seq:
+ * ObjectSpace.allocated_objects => number
+ *
+ * Returns the count of objects allocated since the Ruby interpreter has
+ * started. This number can only increase. To know how many objects are
+ * currently allocated, use ObjectSpace::live_objects
+ */
+static
+VALUE os_allocated_objects(VALUE self)
+{
+#if defined(HAVE_LONG_LONG)
+ return ULL2NUM(allocated_objects);
+#else
+ return ULONG2NUM(allocated_objects);
+#endif
+}
+
+
+
/*
* The GC module provides an interface to Ruby's mark and
* sweep garbage collection mechanism. Some of the underlying methods
@@ -2048,6 +2494,16 @@
rb_define_singleton_method(rb_mGC, "disable", rb_gc_disable, 0);
rb_define_method(rb_mGC, "garbage_collect", rb_gc_start, 0);

+ rb_define_singleton_method(rb_mGC, "enable_stats", rb_gc_enable_stats, 0);
+ rb_define_singleton_method(rb_mGC, "disable_stats", rb_gc_disable_stats, 0);
+ rb_define_singleton_method(rb_mGC, "clear_stats", rb_gc_clear_stats, 0);
+ rb_define_singleton_method(rb_mGC, "allocated_size", rb_gc_allocated_size, 0);
+ rb_define_singleton_method(rb_mGC, "num_allocations", rb_gc_num_allocations, 0);
+ rb_define_singleton_method(rb_mGC, "collections", rb_gc_collections, 0);
+ rb_define_singleton_method(rb_mGC, "time", rb_gc_time, 0);
+ rb_define_singleton_method(rb_mGC, "dump", rb_gc_dump, 0);
+ rb_define_singleton_method(rb_mGC, "log", rb_gc_log, 1);
+
rb_mObSpace = rb_define_module("ObjectSpace");
rb_define_module_function(rb_mObSpace, "each_object", os_each_obj, -1);
rb_define_module_function(rb_mObSpace, "garbage_collect", rb_gc_start, 0);
@@ -2055,6 +2511,8 @@
rb_define_module_function(rb_mObSpace, "remove_finalizer", rm_final, 1);
rb_define_module_function(rb_mObSpace, "finalizers", finals, 0);
rb_define_module_function(rb_mObSpace, "call_finalizer", call_final, 1);
+ rb_define_module_function(rb_mObSpace, "live_objects", os_live_objects, 0);
+ rb_define_module_function(rb_mObSpace, "allocated_objects", os_allocated_objects, 0);

rb_define_module_function(rb_mObSpace, "define_finalizer", define_final, -1);
rb_define_module_function(rb_mObSpace, "undefine_finalizer", undefine_final, 1);

BleakHouse - memory leaks

2009-01-13T17:02:00.000-08:00

As of writing the current version of BleakHouse is 4.1.1.

BleakHouse patches ruby to insert the memory monitoring code. Version 4.1.1 bundles ruby 1.8.6 patch level 230. Unfortunately, attempting to install the gem fails because of a patch failure. Fixing the patching problem leads to double-deallocation and other problems at run-time.

We applied the patches to ruby 1.8.6 patch level 114. The BleakHouse patches change the patchlevel in the version number to be 901.

Quote from acts_as_paranoid not found

2008-12-10T10:05:00.000-08:00

We have been using the acts_as_paranoid plugin and our move to Rails 2.1 (eventually) woke us up to the need to update it.

In case anyone else should wonder why calling delete on their ActiveRecord model instance - which uses acts_as_paranoid - caused a NoMethodFound error on 'quote', it is because your PostGres (or possibly other) database adapter no longer has such a method - although it used to.

The latest version of the plugin fixes the problem.

OpenID

2008-11-17T18:07:00.000-08:00

We see real value to our company in supporting OpenID for our users. Implementation of support as a Relying Party has proved straightforward - thanks to DotNetOpenID.

By supporting registration, as well as login, using OpenID (and only OpenID), we have moved from a typical registration page with many fields to a trivial one. We do have special needs that made this easier but when we realized the simplification this would allow our code and our users we quickly convinced ourselves.

MythTV under Ubuntu Intrepid Ibex

2008-11-17T17:56:00.000-08:00

We have two interesting issues since Intrepid Ibex was released. My updates were tied to the development version but I believe these were observed after the release was final:

MythTV playback sound is higher-pitched in almost every case whether recorded months ago or hours ago. So does live TV.

Lirc was failing to run because it did not see the right lirc device. A soft link in /dev fixed that.

Unrelated to Ubuntu, our remote that came with the multimedia PC case is very flaky with the left and down directions on the keypad. It allows far more than the four compass points. It is possible to recognize the other remote signals but tricky to get the right responsiveness - recognizing all of them produces far too many key presses.

lirc and imon LCD under Ubuntu

2008-08-25T12:32:00.000-07:00

If you have a MythTV installation you will want to control it via a remote control since keyboards and mice rarely fit the decor of a family room.

I chose a Silverstone case with their nice LCD display. It has a potentially rich display. However such sophistication is not supported by the Ubuntu Hardy LCDproc package.

The Ubuntu 8.04 lirc support for the pad on the remote is incomplete also. There is a patch which is targeted for Intrepid Ibex. So I could build from source, add the patch and install. Or I could fetch the Intrepid package and install that. The latter seemed the simpler case but, since it is aiming to be part of Intrepid, its dependencies could be a problem.

Using Intrepid's package as a base fails interacting with udev each time. I suspect that the direct cause is the lack of the lirc_imon driver despite a reference to it in hardware.conf.

dkms is used to manage the lirc support. Unfortunately it believes I have an empty dkms.conf which is causing all the dkms commands to fail.

I commented out the reference to the missing lirc_imon driver and rebooted to see if the poor interaction with udev as fixed.

Rebooting brought up a repair menu, so I chose the option to repair packages. Since I have Intrepid in my sources.list file, it is now downloading 100s of megabytes of updated and new packages for Intrepid alpha 4.

This may take a while to finish... The only upside I can see so far is the chance to become an Ubuntu package maintainer...

Mounting NFS shares on OS X Leopard

2008-06-24T12:35:00.000-07:00

A lot of the information about doing this on Mac OS X refers to the NetInfo database and related utilities, but pre-dates changes which came with Leopard.

On Debian - Ubuntu in my case - most of the information refers to nfs-kernel-server for NFS. Unforntunately nfs-kernel-server requires the uid and gid to be the same on both server and client in the general case. While there is information on how to change your OS X uid/gid it is a pain in the neck and gets worse with more systems.

The best resource I found to address these issues securely is on the Ubuntu forums written by Eiríkr, and uses nfs-user-server to map between the client and server uids and gids - instead of nfs-kernel-server. Install nfs-user-server using your favorite package manager and configure it according to Eirikr.

The Mac mini

2008-06-21T08:12:00.000-07:00

My windows laptop is on it's last legs. I need a machine for development and I loved the MacBook Pro I chose for development at Jackpot Rewards. The only problem I have with the Pro is it's price - it is a premium-priced product.

I have a great 24" monitor on my desk at home and plenty of keyboards and mice. Having a very portable machine is not very important right now. These factors led me to the Mac mini. My current work from home is using Ruby on Rails and my development environment is centered on TextMate so my performance requirements are quite low.

Sure, I would have loved to buy a Mac Pro but the price difference is huge for the extra capabilities. If we get to the point of running an office with a few people I will definitely consider it.

I will write again when i have spent more time using it as a personal system and a development system.

"The Name of the Wind" by Patrick Rothfuss

2008-06-03T13:21:00.000-07:00

I love the Fantasy genre and this book is is one of the best. A great story with well-interleaved flashbacks and great prose.

It was one of those perfect autumn days so common in stories and so rare in the real world. The weather was warm and dry, ideal for ripening a field of wheat or corn. On both sides of the road the trees were changing color. Tall poplars had gone a buttery yellow while the shrubby sumac encroaching on the read was tinged a violent red. Only the old oaks seemed reluctant to give up the summer, and their leaves remained an even mingling of gold and green.

Everything said, you couldn't hope for a nicer day to have a half dozen ex-soldiers with hunting bows relieve you of everything you owned.

The humorous style of writing made it a joy to read. I picked that passage at random - it is the start of the second chapter - but it is typical of the whole. Just reading that passage again makes me want to re-read the whole book. The characters are well-developed and this, the first of three books, finished much too soon (662 pages).

Little Brother by Cory Doctorow

2008-05-09T11:28:00.000-07:00

I was going to write a review of the book but a quick Google gave me Neil Gaiman's review from six months ago. I had read his review but did not try the book because work was just too time-consuming then.

Neil's more recent reference caught me with a little more free time.

Read his review; read the book. Give the book to a teenager you know. I plan to give copies to all the young adults I know.

Spring Configuration

2008-05-09T10:42:00.000-07:00

A recent task was to consume a simple comma-separated file and generate database records from it.

Our application is quite middle of the road in terms of the complexity of it's configuration. One of the most complex parts is the many databases which we access from our code. In this case, I wanted to import test data so we could check out the user interface we were developing for it.

I spent about a week on the task. Four and a half days working on the wiring, trying to find a way to build a configuration which would run and be able to access the database. Just half a day was required to write and debug the actual code.

In the end I gave up on trying to test it in the 'correct' place in the code, and hacked at some code where I knew that I would be able to access the database and have a existing user interface to invoke it from, and see the results.

The code works. Our user interface looks good and displays the imported data.

This configuration overhead adds to development time and makes it harder to justify extensive testing.

We have a number of cross-cutting concerns and use Java 5 annotations extensively. Not all these concerns result in annotations at the places which 'need' them. Our transaction interceptor is an example. For most of our code it 'just works'. The rest of the time it can be hard to know where to start on a solution to the problem.

To be fair, not all of the configuration overhead relates to our Spring wiring. I did lose a couple of hours when a Maven deploy on another branch caused a puzzling regression. That is clearly Maven configuration overhead.

While I have done some work with Groovy, I have not tried Grails yet. I have used Tapestry in the past but Tapestry 5 is a lot different than the Tapestry 3 I used. I doubt that Tapestry will ease these configuration issues. Grails might based on my experience with Ruby on Rails.

Testing by JExample

2008-04-30T10:02:00.000-07:00

The back end of our software - the part which allows us to scale to millions of users - has some very long tests involving networks and databases. They take long enough to run (30+ minutes) that, when run over our VPN, the connection is sometimes dropped before the tests can conclude. While infrequent it is a cause of great frustration when it happens.

Unit tests need isolation from potential side-effects. Providing that isolation requires potentially extensive and / or time-consuming setup and sometimes tear-down code.

A very new testing framework based on JUnit is called JExample. Unlike JUnit, tests can depend one upon another. Since the tests have passed, the environment is in a known, tested as correct, state and can be relied upon as the basis of subsequent tests.If test A passes, then test B can rely upon the result and context of A. When A is a long test, or requires considerable setup or teardown, this can result in substantial time savings.

A dependency graph of tests is built and used to order the tests.

The main results are less elapsed time for the tests and test failures which are more likely to pin-[point the exact place where a problem is occurring.

The work is based on the paper by Kuhn, Van Rompaey, Haesenbarger, Nierstrasz, Demeyer, Gaelli and Van Leemput. There is an introduction and Eclipse integration. Java 6 is required.

The Rails Way

2008-04-29T14:56:00.000-07:00

I never picked up my novel. The Rails Way is well-written enough to have kept me reading it through the whole vacation. I read about 500 pages and if our vacation had been any less busy I would have finished it.

The text is easy to follow no matter how complex the topic, with the exception of a passage relating to meta-programming. I will have to go back and see if I do better a second time around.

I found a few typos and errors but none that obscured my understanding of the author's true intent.

The testing section had some good coverage of Rail's built-in facilities and also of RSpec. I have been reading about Shoulda recently also and I plan to compare the two more closely.

Agile Web Development with Rails has been considered the leading book on Rails. For a newcomer to the topic it is still a better book but, for anyone with some knowledge of the framework, this is a much better organized volume. Sam Ruby is updating it for Rails 2 but it will only be a beta book for most of this year.

Hill towns of the Cote d'Azur

2008-04-29T05:29:00.000-07:00

Haut de Cagnes and St. Paul de Vence were two hill towns which we visited. People have been seeking the shelter of these steep hills since at least Roman times.

They both have associations with the Grimaldi family - rulers of Monaco - but only Haut de Cagnes attained the importance to gain a castle.

As a child I read how Hannibal crossed the Alps with his Elephants to avoid a Roman army blocking the route to Italy. Given the terrain it would have been very hard to force a passage along the coast with plenty of places for the Romans to set up a strong defensive position.

Now the coastal hills are thickly littered with houses showing how many people are attracted to the scenery, climate and food.

Haut de Vance
town square from
the castle roof.

2008-04-24T13:58:00.000-07:00

From the Left Bank to Notre Dame to the Louvre along the Champs Elysee to the top of the Arc de Triomphe to the Eiffel Tower to a bus back was about six miles including the long, long corridors of the Louvre. Excellent exercise after months of keyboard pounding.

Arc de Triomphe

I took photographs of each of the twelve roads leading from the Arc de Triomphe.

Notre Dame has a crypt but, here, that is an archaeological site showing some of the foundations and wall upon which the area in front of the cathedral has been built. Apart from the many changes in use - including as houses and two different hospitals - it was a shore fortification in Roman times although the bank of the Seine is now at least fifty yards distant.

Do I hear footsteps?

2008-04-24T10:02:00.000-07:00

A friend lent us a map of Paris. After arrival we noticed that she had underlined the metro station we used on the way in from the airport. She had also underlined the road of our hotel. The road is only 100 yards long and just has one hotel.

Before looking at the map I would have imagined that the odds of us staying at the same hotel were small. Now I have to believe that we actually did stay there...

2008-04-23T13:50:00.000-07:00

The Rails Way is proving very readable and I am yet to pick up my novel.

Paris was fantastic - more on that later. We are now staying in a converted castle guardhouse near Nice. The restaurant here is incredibly good. Our room is huge with a balcony looking out over the valley. There are some very acceptable frog noises in the background as type. We are planning a more relaxing day tomorrow so I will catch up on the events of Paris then.

Vacation reading

2008-04-19T14:04:00.000-07:00

I often find it difficult to decide what books to read on vacation. Fun? Technical? If I really think I need to read some thick technical tome, I will often take just that one book. If I take the big book and a fun one, I know which one I will pick up when I get a moment.

For this trip I brought along "The Rails Way". I have found it very readable so far. I have been using Rails for a couple of years now and I find the approach of this book much more suitable now than the "Agile" book.

At the risk of not reading The Rails way, I also brought along Shadowplay by Tad Williams. I am a huge fan of his other books. Memory, Sorrow and Thorn is from the same genre but I liked it much more than the first book in this series. I will see if it grows on me...

The plane is waiting...

Making a start

2008-04-18T10:25:00.000-07:00

My last vacation was to Scotland in 2006. I love the Highlands and Islands: The Great Glen, Skye, Kyle of Lochalsh, Mallaig, Oban and the lochs and valleys which connect them. Tobermory on Mull is a great favourite.

Tomorrow we leave for Paris and Nice. They will be such a contrast. One of the great cities of the world with its monuments and museums; and the grassy, rocky, lochs of the Scottish coast. Each is beautiful in its own way - each refreshing to the soul.

I still have to decide whether to bring my MacBook Pro with me. I can not remember the last vacation I took where I did not spend hours working late at night after a busy day's sightseeing. I have some ideas for a new web site. It would be a contrast to the heavy Java coding of the last ten months. I would probably use Ruby on Rails although Merb is an interesting and tempting alternative.

If I do not take the laptop with me, this first post might not be succeeded for a couple of weeks. I would love to be able to add some photos of Paris though...