Top 10 Runtime Power Management mistakes

Since I had more than one person ask if I could document this somewhere, here is my part of the presentation that Len and I did for LPC2011.  (yes, as requested I will also be submitting this as a patch to Documentation at some point soon)

This past year I was tasked with enabling runtime power management for most of the drivers for the Moorestown architecture for Meego.  I was also the Meego kernel maintainer for the Moorestown architecture for a few months, and got to review many drivers during that time.  Below are the top 10 issues I found driver writers making (over and over and over again) while attempting to implement runtime power management.

  1. Not understanding the general concept of runtime power management.  Runtime power management is initiated by the kernel and transparent to user space.  YOU determine when you are not busy and sleep then, you don’t wait for someone to tell you to.
  2. Not understanding what your subsystem does for runtime PM.  I saw many driver writers duplicating work that is done in their subsystem core.
  3. Make sure you understand all the possible entry points to your driver.  Sysfs is an entry point.
  4. Make sure you understand when you are actually touching hardware.
  5. Make sure you understand the context in which you are being called during these entry points (i.e. are you called in interrupt context?).
  6. For drivers who may have entry points called in interrupt context, you must ensure that the hardware is available without sleeping, otherwise use a work queue.
  7. Ref counting during the irq handler does cost cycles.  This may be ok, depending on your device.  If you have to do ref counting during the irq handler, use a threaded irq handler if possible (see #6).
  8. Have the correct granularity for your ref counting.  Depending on your hardware, there is likely some kind of performance penalty for entering a suspended state.  You many even lose device state.  Restoring can take time, so you want to make sure you are appropriately idle before allowing a suspend.
  9. Unbalanced ref counting occurs when you do not pay attention to error paths.
  10. Unbalanced ref counting can occur when you try to do something tricky with the ref counting to work around overly complex code paths (like setting your pm usage count directly to zero).  This will cost you hours and hours of debug time.  If it isn’t obvious where to increment/decrement your counter, you are doing something wrong.


Site upgrade (finally).  Hopefully this will stop this site from getting hijacked.  Lost a few posts already – I am a failure at website admin.  Good thing this isn’t my day job!

To Brine or not to Brine

Last year we raised 15 heritage turkeys – the “rare turkey assortment” from McMurray Hatchery. We lost 1 the night before the big trip to the processor, and saved 3 in an ill-fated attempt to keep a few for breeding.  Of the 11 we took to the processor, I sold 7 of them to friends and kept 4 for myself.  I ate 2 of them last year, and have 2 left.  Letting your meat sit a year in the freezer isn’t generally a good idea, but I knew in the spring that we were not going to get any fresh ones this year so I went ahead and saved one to cook for this thanksgiving.

Commercial turkeys are almost always a breed called the Broad Breasted White.  These birds have been raised mostly since after WWII, and were engineered to have put on the maximum amount of weight in the shortest amount of time.  They also have white feathers, which make it easier for people not to notice when there are pin feathers left in the bird after plucking.  These birds get gigantic – and they do it in record time.  They do such a good job at getting big, that they get too big to do the deed, and without the friendly hand of man (ok, who likes that visual 🙂 ) would not be able to reproduce.  They also can’t really walk around that much either – so even if you buy a free range turkey, if it was a broad breasted white chances are it didn’t range far from the feeder.  This makes for some soft mushy muscles – not unlike what you wind up with when you sit in front of a computer all day, only moving to feed yourself.  These turkeys don’t get very old either – they can reach market weight in a little more than 4 months.  Young birds just don’t get a chance to develop much flavor.  But that’s ok – these birds were not bred for flavor, they were bred for cost effectiveness.  That’s why a lot of these turkeys are sold injected with stuff to make them taste better.  Well – I guess that injecting also makes them heavier, so people can charge you for water weight.

By contrast, heritage breeds are a more self-sufficient bird.  They do not get as large as a commercial turkey breed, and as a result have no problems reproducing without our help.  They have not been bred to have DDD size breasts, and they can fly.  They also have better immune systems.  They take longer to reach their market weight, so they’ll be older when they go to the processor – usually at 6-7 months.  This means they have some time to develop more flavor.  They will range easily, and are happiest when they can eat bugs and grass and what not.  They aren’t as food obsessed, so will leave the feeder and go roaming.  This means that they get a lot of exercise, which makes them lean and gives their meat more texture.

So with the big day arriving tomorrow the question of how to cook this fancy heritage turkey has come up again.  If you google “Cooking Heritage Turkeys”, you’ll get a lot of very strong opinions on the subject.  The one thing people agree on is that you probably don’t want to treat this turkey the same way as your standard commercially raised turkey. Two of the biggest things I see people discussing are whether or not you want to brine this heritage turkey, and what type of cooking temperature to use.  Last year I followed the recommendations that I read here and went with the no-brine, high temperature approach.  I pulled it out when the thigh reached 150 for fear of overcooking, just as the article suggested.  The white meat was absolutely perfect and delicious.  The dark meat was undercooked and tough (but still really tasty – I ate it anyway).  Over the summer, I decided to try a different cooking method, and I brined the turkey and smoked it – so the low and slow method.  While the skin was like leather, the meat itself was delicious, and the dark meat was much better cooked this way.  So – I’ve decided that for this Thanksgiving, I’m going to do the brining again and roast it at a lower temperature – I’ll probably do 275.  I’m also going to let the thing come up to a higher temperature.  I found one recommendation for 180 at the thigh, but I think I’ll take it to just 165 and see how that goes.  The recipe I’m using is below – I’ll post a review of how it turned out when I’m done.

To prepare the turkey:

  • Put 1 8 pound heritage turkey in a 5 gallon stockpot with 3.5 gallons water.  Add 1 cup kosher salt per gallon of water, and 1/4 cup maple sugar per gallon water (I ran out of maple syrup, so I used mostly maple syrup and a little bit of honey to get me to the right amount).  Stir up salt till it dissolves, cover pot and put in fridge for 12 hours.  Make sure the turkey is fully submerged.  Remove from brine and allow to air dry in the fridge for 8 hours before cooking.
  • Create an herbed butter mixture with about a pound of butter, whatever fresh herbs I have laying around the yard (which happens to be sage, parsley, and rosemary) and some pepper.  Rub this mixture under the skin as well as on top of it.  Put some in the cavity too.
  • Stuff cavity with onions, celery, and carrots.
  • Put oven at 275 and roast, breast side up till close to correct internal temp (165 at thigh).  I’m guessing this will take a good 4 hours.  Then crank the oven to 450 and brown for about 20-30 minutes to crispy up the skin.

I’m still internally debating whether to mess with starting the turkey breast down and then flipping it, or whether to brown first at 450, then drop the temp, or do the browning last.  I am pretty sure I’m sticking to the plan listed above, but you never know what I’ll do in the heat of the moment. Flipping a bird (heh) is a much more doable thing when your bird is only 8 pounds – I definitely wouldn’t even consider it for one of those giant birds.

Mission Accomplished

for real.

I couldn’t have been happier with how the Linux Plumbers Conference went last week. I went back and looked at the original proposal that we had Arjan, Greg, and Randy present to the Linux Foundation, and we seem to have hit all our original goals. From conception we wanted this to be a “working” conference – and from the conversations in the hallways that I overheard, to the discussions in the microconfs that went on, I could see that people were indeed getting together, discussing issues and solving problems. Conferences require a lot of time, effort, and money to do right, and it’s gratifying to feel that something useful will come out of this.

I think that now I can go back to blogging about duck poo and vegetables.

What is a Plumbers Conference?

After spending a few days manning the Plumbers booth at OSCON, I thought I’d post the answer to the question that everyone seemed to want to know – What is a Linux Plumbers Conference? We came up with the word “Plumbing” to describe the low level infrastructure of a Linux System. This includes the Kernel, desktop infrastructure like X and graphics libraries, system utilities like udev and hal, as well as essential libraries like glibc and friends. These components interface with each other at times – some better than others. We hope to provide a forum for people from these types of projects to get together and try to solve problems that are system wide or cross multiple project boundaries.

In addition to the topics to be discussed in the microconfs and the general talks (see, we will have “unconference” style talks. We have several smaller rooms available for people to get together and work out specifics, talk about something they didn’t get on the schedule, or have a group hug. These rooms can be reserved at the start of the conference.

August 18th the registration fee for Plumbers will increase to $300. If you haven’t already registered, what are you waiting for?

Appleyard broodiness and garden progress

It was an eventful week in the Appleyard universe.  The Silver Appleyards are a heritage duck that was popular in the 30’s.  Almost nobody breeds them anymore, and as such they are listed as critical by the ALBC.  They are quite large, grow fast, and are fairly regular egg layers, so good for both meat and eggs.  I bought a breeding pair last year from the Holderreads, one of the world’s most respected duck breeders, and lucky for me, within driving distance of here.  I tried to name them Bob and Alice, in typical geek fashion, but Kyle insisted on renaming the male “Bill”, which did suit I suppose.  Last year I put 4 of their eggs under a broody chicken and hatched 3 more females and 1 more male.  This winter we lost Bill to probably an owl, so Bill’s son became the new Bill.  A couple weeks ago Alice got a nice nest built up and began sitting on it, but sadly this past weekend we lost her to an owl as well.  I’ve started locking the remaining Appleyards up at night – which has been surprisingly easy.  But, Tuffy – another Appleyard female – got broody about a week after Alice, and before the Owl Incident, and is sitting on a nest so buried in the brush that we can’t get a fence around her.  Unlike a chicken, you can’t move a broody duck – so if we move her we have to accept that she won’t be hatching any eggs.  Considering the loss of Bill and Alice, we really want Tuffy to succeed, so we are going to leave her there and hope for the best.  Tuffy got her name because she was born with a non-breed-conforming tuft on the topish of her head. Ideally, we wouldn’t breed her. But, from my perspective, she appears to have a favorable trait that I’d like to keep – the desire to sit on eggs and make more ducks. So, despite her lack of conformation, we will probably save some of her young and allow them to breed next year. I guess everyone’s got their own priorities when it comes to breeds. My priorities for the ducks are: large, tasty, generally takes care of themselves, and can breed naturally.  The other Appleyard females are now laying in the pen at night, so I will probably start letting a nest accumulate in there in hopes of convincing another one to set.   I’m also going to set a “broody chicken trap” in the chicken coop and see if I can’t catch one of them.  I just pile up a bunch of fake eggs in one of the nests and then if a chicken has the broody instinct, it might just convince them that it’s time to get serious.  We’ll see what happens.
I got out into the garden last week for a few hours. Planted a bunch of potatoes and a few leeks. I am running out of room in the upper garden. I might have to expand. The weeds are a nightmare right now. I’ve not been up there since last year. The blueberries are simply buried in weeds. I spent about 3 hours weeding the blueberries and only got about 1/2 way done. The peas I planted in February really desperately need weeding – but there was no time.

Animal, Vegetable, Miracle followup

I finally finished reading Animal, Vegetable, Miracle during a recent trip, and I am feeling much less depressed about it than I was only 20 pages into it.  What is revealed as you read the book is that Barbara and her family have been very serious gardeners/canners/poultry raisers for years prior to this.  They spent a whole year remodelling their farmhouse first before they embarked on their quest to feed themselves.  Their barns were already built.  And they’d been building their soils and their gardens for many years on this property.  They had well established asparagus beds, and knew exactly how much they had to grow of what crops and how they had to grow it.  If I’m reading the book correctly, it looks like Barbara had about 15 years of experience prior to this point to practice.  So now I don’t feel so bad, and in fact I think I’m doing pretty good considering I’m only in my third year of practice.  It’s hard to be patient when you have such a clear vision of what you want to do.  Unfortunately, that vision also shows me all the work it will take to get there.  Patience, it seems, is my only option.

Pesto for early spring

Today was the first weekend that I’ve been home in over a month that has had decent weather. I spent the day weeding the lower garden and trying to get caught up on some planting. I planted Chiogga beets, early cabbage, broccoli, a spring lettuce mix and some bok choy. I also harvested some of the spinach that I’d planted in the fall. Planting Spinach in the fall works out really well – even though you can’t really eat it by the time winter comes, it stays alive all winter and gets a head start, so there’s actually something growing in the garden in April that is ready to eat. I invented a pesto recipe to use up the spinach – it’s a seasonally appropriate pesto, so no basil in it.

4-6 cups washed Spinach

2-5 cloves of garlic

1/2 – 3/4 cup olive oil

1/2 to 1 cup of walnuts

1/4 tsp salt, and pepper to taste

Put all ingredients in food processor and process till smooth. toss with Pasta and crumbled feta cheese.

In other news, Mrs. Whitey the white Muscovy duck decided to get broody again while I was out of town. She decided to put her new nest inside the chicken coop, underneath the nest boxes. This is a pretty safe place as far as I can tell – the biggest danger is that chickens will decide to hide some eggs on her nest. Chickens will hatch in only 21 days, but Muscovy ducks take 35 days. Alice the Appleyard has got a nice nest going too – although up to now she’s not shown any signs of wanting to sit on it.