3 03 2009

Sorry it’s been so long … going back to school after 8 months of life in the real world is a bit of a system shock. Imagine, working on interesting problems with smart people and constant learning, only to come back to an environment where “the answer the prof wants” is the only right answer, and there’s never any rush to get anything done. This kind of makes one get lazy.
Now that I’m through the first half of the semester, it’s time to pick up my socks and get more work done!
Once again, I took a look at what I’d done to our copy of DTrace and said to myself “God, what a hack job!” It’s time to step back again and do this right. I asked Colin the other day if he could give me any examples of projects under out source tree that contain makefiles to produce multiple binaries. esh was the prime suspect for that; now it’s a matter of studying the common.mk to formulate a decent one for dtrace to do a similar job. Unfortunately, I still not all that familiar with makefiles, and makefile magic, so it’ll take awhile.
Anyways, I’m off to my next class. Toodles!

Baby Jesus was born todaY!

24 12 2008

Merry Christmas!

I am writing this from home on the rock. After a fantastic eight months of working for QNX software on a co-op term, I’m back in Newfoundland with my fiancee, and my school buddies, and before long I’ll have my nose back to the grindstone. I have to apologize for not writing much recently, my last week or so at work were fairly hectic; who would’ve thunk that there was so much work to do just to get out of a workplace?

Anyways, now that we’re over with that, I’ve got good news, and I’ve got bad news. The good news is that I’ve gotten all of the CTF utilities to build, the bad news is that some of them still have runtime issues. In other bad news, the past 2 weeks have been nuts, and I haven’t gotten any real work done on the utilities. On my last day of work, however, I managed to get all my source code up on the foundry, so now I can work on it … but not just me, you, yes, even you, can now download and work on the DTrace for QNX project … although, the source code was available before, there’s just more now!

That’s it for now! I’ll post results when I get more work done.


When object linkers are outlawed, only outlaws will have object linkers …

5 12 2008

It’s been a few days since any updates, sorry. I wanted to wait until I had something exciting to share, but it look like that might be a few more days.

The CTF utils are coming along, although, not as quickly as I’d like them to be. I’m working hard on them, but my limited knowledge about the way the internals of our build system works is throwing me off a little bit. This morning I had Colin come over to my cube and explain to me why I had so many “undefined references” to functions that I was sure were linked in. It only took a few moments for him to figure it out, where it took me most of the afternoon yesterday to come up blank. Apparently, when you statically link a library into another library, it basically just gets a container of the library’s object files. Let’s use an example to make this easier. Let’s say I have foo and bar in my build tree; foo is my library, that I want  to build so I can link to it from an application, and bar is a pre-built library that contains some of the functionality I need for libfoo. If I were to add “LIBS += barS” to my common makefile, and then run “make spotless hinstall install” (under the QNX toolset), then libbarS.a, which is merely a container holding all the object files from libbar, I would have a complete libfoo.so. If I were to then try to include libfoo into my application, let’s call the application “util,” then I would add

LIBS += foo


LIBS += fooS

to my common makefile for util. If I were to then run make for util, ld would error out because it can’t see the symbols that libfoo requires from libbar. This is because ld can’t see the *.a container contained within libfooS.a, so in order to statically link the libraries to the application, I would have to add

LIBS += barS fooS

to my common makefile, either that, or let the linker find the symbols automagically from the shared objects by using

LIBS += bar foo

The greatest surprise for me about this whole endeavour was not only that the static libraries inside of a library can’t be referenced by ld, but rather, that ld can’t handle linkage dependencies. Someday I would like to see ld be able to link a library into an application, and have all the libraries that are linked to that library be automatically linked. For example, if I had libfoo1 libfoo2 libfoo3, all of which are dependencies for libbar1, and an application foobar, I want to see make/ld be able to link libfoo* to libbar1, and then only be required to link libbar into foobar. All the information that’s required by the linker to do this at compile/link time should already be provided by the library being linked into foobar. Who knows, if I get enough time, eventually I may be able to make those changes, for now, though, I’m just going to sit back and complain.

I’ll post again when I’ve got more to say.

I shall call it the “wheel”

28 11 2008

Well, I wasn’t on top of my game yesterday. Of all things, I had put the wrong copy of the DTrace source onto my iPod to bring to work! The network had been a little wonky after my last post, so I figured I would use my iPod as a disk and bring the source back to work the good old fashioned way! Little did I realize that there were two copies of the source on my machine at home, and I took the wrong one, d’oh! With all mistakes, however, there comes a lesson; I don’t know what the lesson for this one was, but I got to use another lesson I’d learned ;)

I grabbed the source again, and instead of getting things all mixed up, I separated it cleanly, and used addvariant to create the build structure, and fixed a few missing pieces from the solaris_compat lib that I’d added to at home. To be a little more clear on what I mean by “cleanly” : I removed all the code common to the three utilities from the ctf directory, and made a lib called ctfutil, which each utility will link to statically, then I created project directories for each of ctfdump, ctfconvert, and ctfmerge. It took me about an hour to recreate the source tree, with a more logical separation, and get a package that builds correctly. Now, thanks to my extensive practice at creating the build trees and adding to libsolaris_compat for ctfutils, we have a working ctfdump utility! Of course, I don’t have an opensolaris install on hand at the moment, nor do I have any binaries that I’ve compiled with embedded CTF data, so I popped in the old OpenSolaris 2008.11rc1 install CD and grabbed the first binary I could find (and crossed my fingers, hoping that it had embedded CTF information!), $(CD_ROOT)/platform/i86pc/kernel/unix (the kernel … that’s got to have *something* interesting in it, right?). The next obvious step was to try out the ctfdump utility from my neutrino vm, and the output looks a little something like this:

– CTF Header —————————————————————–

cth_magic    = 0xcff1
cth_version  = 2
cth_flags    = 0x01
cth_parlabel = snv_101a
cth_parname  = genunix
cth_lbloff   = 0
cth_objtoff  = 8
cth_funcoff  = 3522
cth_typeoff  = 25448
cth_stroff   = 74256
cth_strlen   = 48023

– Label Table —————————————————————-

34357 snv_101a

– Data Objects —————————————————————

[0] 1549     cbe_low_hdl (97)
[1] 1549     cbe_clock_hdl (98)
[2] 18       cbe_ticks (101)
[3] 33229    cbe_xcall_func (102)
[4] 33230    cbe_xcall_cpu (103)
[5] 32781    cbe_enabled (104)
[6] 37       cbe_xcall_farg (105)
[7] 18       cbe_vector (106)
[8] 199      cbe_timer_resolution (107)
[9] 33120    cmi_list (125)
[10] 40      cmi_load_lock (126)
[11] 33153   cmi_hdl_ops (164)
[12] 62      cmi_pcicfg_flags (166)
[13] 62      injcnt (169)
[14] 33166   msrihash (170)

– CTF Statistics ————————————————————-

total number of data objects        = 1757

total number of functions           = 2415
total number of function arguments  = 4669
maximum argument list length        = 9
average argument list length        = 1.93

total number of types               = 1589
total number of integers            = 2
total number of floats              = 0
total number of pointers            = 313
total number of arrays              = 208
total number of func types          = 0
total number of structs             = 498
total number of unions              = 31
total number of enums               = 25
total number of forward tags        = 3
total number of typedefs            = 475
total number of volatile types      = 3
total number of const types         = 31
total number of restrict types      = 0
total number of unknowns (holes)    = 0

total number of struct members      = 3767
maximum number of struct members    = 80
total size of all structs           = 75907
maximum size of a struct            = 9520
average number of struct members    = 7.56
average size of a struct            = 152.42

total number of union members       = 136
maximum number of union members     = 25
total size of all unions            = 9936
maximum size of a union             = 8192
average number of union members     = 4.39
average size of a union             = 320.52

total number of enum members        = 343
maximum number of enum members      = 114
average number of enum members      = 13.72

total number of unique strings      = 4092
bytes of string data                = 48023
maximum string length               = 33
average string length               = 11.74

is it just me, or is this not AWESOME?

So, what I found next is that the output of “du unix.ctf” was about 9600, and that the total size of the unix executable was about 20000. Don’t Panic! This doesn’t mean that CTF is *big*, just that it stores a lot of data; I don’t have numbers yet, but when I get ctfconvert and ctfmerge working, I’ll post some comparisons between executables with and without embedded CTF data. To give you an idea of how much information was in the unix.ctf file, “cat unix.ctf | wc -l” reported 14,940 lines! That’s a lot of data!

So, not much more on that for now, but we’ve got one utility working, two more to go!

Are we there yet?

26 11 2008

as this morning rolled around I had a fresh whole day to myself, to do work. One of the benefits of being but a lowly programmer is that I’m not necessarily required to be at work for all the standard work hours, meetings excluded. I decided that today I would work from home, and got the go-ahead from my boss. I was surprisingly productive. The last time I worked from home, I spent a lot of time just trying to set up my dev environment on my home computer (my workstation at work is just that, a workstation; it’s not very mobile).

The project on which I am currently working is a QNX port of the DTrace utility and framework. The person I’m working under for this project is no less than the famous Colin Burgess of the SendReceiveReply blog … also a legendary kernel developer for the QNX operating system. Colin tells me what he wants done for the DTrace effort, and I do my best to carry that out. The project hasn’t gotten much attention for awhile because of the recent release, which was quite time intensive.

If you haven’t seen what DTrace can do, I suggest you watch Brian Cantrill on google, grab a copy of FreeBSD or Opensolaris, and have at ‘er!

I’m not going to go too far into the technical details of what DTrace can, or can not, do at the moment; well, maybe I will tell you that DTrace can’t pour a decent beer, but I will enlighten you on my progress with the port thus far. There is a porting status page on the foundry, which, to date, will show you all of the work that Colin has put into the project. At the moment I’m working on the CTF utilities for QNX so that we can actually convert dwarf debugging information into Sun’s CTF (Compact C Type Format) debugging information natively. Currently we need to use Opensolaris or FreeBSD to run the utilities to get the CTF code into our binaries.

The first bit of work that I had to do to get the ctf utils to compile in a QNX build tree, was to “munge” the source into a QNX style source tree. That was something which I accomplished, with some difficulty last week. Only today as I was reading the documentation for our addvariant utility did I realize that I didn’t have to do any of that work manually at all! The utility will take GNU style build systems and squash them, create the required files and directory structure for a QNX-style make build, and voila! No more manual munging! Fortunately I had a chance to use this newfound information today. The ctfutils require libdwarf, a library for processing dwarf information embedded in binaries. There was a libdwarf.cpio.bz2 archive located in the ctf directory (the project level dir containing the source code for all 3 ctf utils), so, like a good little programmer, I unpacked this and assumed it was the libdwarf I was supposed to use … that was last week. I spent several days trying to debug issues of missing symbols and such. Today I ran into a symbol that was nowhere to be found in the source I had available, except for where it was being referenced. I did, however, stumble accross the symbol in a google search, accidentally, on a freebsd mailing list. It turns out that the library the utils use is the freebsd libdwarf implementation. This juicy little tidbit helped greatly! Replacing the libdwarf gave me the opportunity to move onto other problems, and now we have utilities that actually compile (with some kludges), woo-hoo!

Well, tomorrow I get to test my changes on the neutrino box I have at work.

Until next time.

Hello world!

25 11 2008

Hello world! This a simple shout out to say that I’m alive, and well … if you’d prefer the hex representation of the ascii text, I could just say: 0x48 0x65 0x6c 0x6c 0x6f 0x20 0x77 0x6f 0x72 0x6c 0x64 0x21 0x00

or, in binary (think: 13 element array of unsigned char): 01001000 01100101 01101100 01101100 01101111 00100000 01110111 01101111 01110010 01101100 0110 0100 0100001 00000000

now that I’ve wasted enough time: this blog is simply going to outline some of my thoughts and endeavours as I finish up a co-op with QNX Software Systems, and embark on my final year as an undergrad computer science student. Hopefully I won’t be the only one learning anything :)