AndrewHeiss.com

Install R, RStudio, and R Commander in Windows and OS X

2012-04-17T00:00:00-06:00

R is an incredibly powerful open source program for statistics and graphics. It can run on pretty much any computer and has a very active and friendly support community online. Graphics created by R are extremely extensible and are used in high level publications like the New York Times (as explained by this former NYT infographic designer).

RStudio is an integrated development environment (IDE) for R. It’s basically a nice front-end for R, giving you a console, a scripting window, a graphics window, and an R workspace, among other options.

R Commander is a basic graphical user interface (GUI) for R. It provides a series of menus that allow you to run lots of statistic tests and create graphics without typing a line of code. More advanced features of R aren’t accessible through R Commander, but you can use it for the majority of your statistics. (Lots of people (like me) use R Commander as a crutch for a few months before they get the hang of the R language. As intimidating as it might be to constantly type stuff at the console, it really is a lot faster.)

However, as is the case with lots of free and open source software, it can be a little tricky to install all of these different programs and get them to work nicely together. The simple instructions below explain how to get everything working right.

Install R, RStudio, and R Commander in Windows

Download R from http://cran.us.r-project.org/ (click on “Download R for Windows” > “base” > “Download R 2.x.x for Windows”)
Install R. Leave all default settings in the installation options.
Download RStudio from http://rstudio.org/download/desktop and install it. Leave all default settings in the installation options.
Open RStudio.
Go to the “Packages” tab and click on “Install Packages”. The first time you’ll do this you’ll be prompted to choose a CRAN mirror. R will download all necessary files from the server you select here. Choose the location closest to you (probably “USA CA 1” or “USA CA 2”, which are housed at UC Berkeley and UCLA, respectively).
Start typing “Rcmdr” until you see it appear in a list. Select the first option (or finish typing Rcmdr), ensure that “Install dependencies” is checked, and click “Install”.
Wait while all the parts of the R Commander package are installed.
If you get permission errors while installing packages, close R Studio and reopen it with administrator privileges.

Install R, RStudio, and R Commander in Mac OS X

Download R from http://cran.us.r-project.org/ (click on “Download R for Mac OS X” > “R-2.x.x.pkg (latest version)”)
Install R.
Download RStudio from http://rstudio.org/download/desktop.
Install RStudio by dragging the application icon to your Applications folder.
Download Tcl/Tk from http://cran.r-project.org/bin/macosx/tools/ (click on tcltk-8.x.x-x11.dmg; OS X needs this to run R Commander.)
Install Tcl/Tk.
Go to your Applications folder and find a folder named Utilities. Verify that you have a program named “X11” there. If not, go to http://xquartz.macosforge.org/ and download and install the latest version of XQuartz.
Open RStudio.
Go to the “Packages” tab and click on “Install Packages”. The first time you’ll do this you’ll be prompted to choose a CRAN mirror. R will download all necessary files from the server you select here. Choose the location closest to you (probably “USA CA 1” or “USA CA 2”, which are housed at UC Berkeley and UCLA, respectively).
Start typing “Rcmdr” until you see it appear in a list. Select the first option (or finish typing Rcmdr), ensure that “Install dependencies” is checked, and click “Install”.
Wait while all the parts of the R Commander package are installed.

Open R Commander in Windows and OS X

Once you’ve installed R Commander, you won’t have to go through all those steps again! Running R Commander from this point on is simple—follow the instructions below.

If you decide to stop using R Commander and just stick with R, all you ever need to do is open RStudio—even simpler!

Open R Studio
In the console, type windows() if using Windows, quartz() if using Mac OS X. (This tells R Commander to output all graphs to a new window). If you don’t do this, R Commander graphs will be output to the graphics window in RStudio.
Go to the “Packages” tab, scroll down to “Rcmdr,” and check the box to load the plugin. (Alternatively, type library(Rcmdr) at the console.)

World-Ready Composer not Perfect

2011-06-25T00:00:00-06:00

Although Adobe has included the world-ready composer in InDesign CS4 and 5, like I said in my previous post, it’s not documented or supported at all. It’s still buggy and unfinished, unfortunately.

Here’s a little example of how buggy it really is. I typeset the same sentence in Arabic in different fonts I have installed on my computer. As you can see, most of the fonts work flawlessly (yay!), with two exceptions. Traditional Arabic can’t display short vowels—they break up the connecting letters—and Geeza Pro is a sad, sad little font.

Download PDF of sample

I don’t know if this is a problem with the fonts themselves or with the composer. Traditional Arabic was specially designed for Office 2007 to be a high quality Arabic font, and it comes with thousands of extra glyphs for every position possible, so it would seem like it *should* be able to display correctly. Just kidding. Arabic Typesetting is the special Office 2007 font and it works just fine (it has a nice Naskh-y feel to it). Traditional Arabic has been included with Windows since Windows 2000 and apparently isn’t very well made (?). Or maybe the composer is just buggy (?).

Geeza Pro, on the other hand, definitely has a problem with the font itself. This forum says that the version of Geeza Pro in Snow Leopard is faulty and doesn’t connect. Hopefully Lion fixes that…

Using Arabic in InDesign CS5 without InDesign ME

2011-06-24T00:00:00-06:00

Nearly four years ago, I posted a workaround for InDesign CS3’s lack of support for RTL support. The only way to get proper Arabic or Hebrew text back then was to type backwards—something trivial (and scriptable) for Hebrew, but far more complicated for Arabic, with all its different letter forms that change depending on their position in the word. Rather than type backwards, you had to manually insert glyphs from the glyphs panel backwards. While this was extraordinarily tedious, it worked for times when you only needed to deal with a few words—maybe a short sentence.

For my job I’ve been using InDesign CS3 ME with Tasmeem and have fallen in love with the super advanced typographic tools that it provides. But, I’m not planning on keeping that job forever (I’m graduating in less than a year!), and I don’t want to go back to hunting for glyphs. I’ve been spoiled :)

I heard rumors that CS4 and CS5 had a mysterious RTL editing mode that was only accessible through scripting, but I was in Egypt for both launches and wasn’t able to get the educational upgrade. I finally upgraded to CS5 a couple days ago and immediately got to work figuring out this rumored Arabic mode.

Starting with CS4, Adobe began including a new “world ready” paragraph composer, which provides many of the same typographic controls offered by Winsoft’s Middle Eastern editions. However, CS4 shipped before Adobe could polish off any of the controls. The internet world hoped that there’d be some new world ready panel for CS5, but to no avail. The world ready composer was included (and improved?) with CS5, but is still mostly inaccessible.

Unless you do a cool trick :)

You can programmatically create paragraph styles that use the world ready composer, allowing you to typeset Arabic, Hebrew, Thai, Devangari-like languages, and a ton of other more complex scripts. Here’s how you can enable native Arabic typography inside CS5 (courtesy of Thomas Phinney):

Download these scripts by Thomas Phinney and Peter Kahrel and install them in your script folder.
Run the “r2l Paragraph Style Arabic” script from the Scripts panel. A new paragraph style named “RTL Arabic” should appear in your Paragraph Styles panel.
Check the style settings and verify that the “Adobe World-Ready Paragraph Composer” is being used for the style.
Start using Arabic text!
Bonus: To use the composer in other documents you can either run the script again or just copy the style. The composer is activated by the style. Theoretically you could save the Arabic style to a new template and never have to run the script again.

This is definitely not a full replacement for Winsoft’s CS5 InDesign ME, since it doesn’t give you any graphical control over kashidas, digits, diacritic positions, or any of the other more detailed options available in the world ready composer. You can edit the script, change the options around, and create a new style, but you’d need to know the API for the world ready composer. Phinney’s script, for example, sets the default digits for the style as Arabic digits with ps.digitsType = DigitsTypeOptions.arabicDigits;. If you knew the option for Farsi-style numbers, you could edit the script. Unfortunately the whole composer is completely undocumented and unsupported. This site includes many of the hidden options—you’d just need to guess about how to use them in the script. Maybe Adobe will finally make all these options accessible in CS6?

There are a couple options for more graphical control over the hidden settings, but they cost money. idRTL created a plugin that provides a useful panel to edit all the hidden Middle Eastern settings. It’s $50 and says it’s made for CS4, although I’m assuming it’ll work with CS5 too. Word Tools creates a similar panel, costs $100, and supports CS5.

So, even though activating the world ready composer can be a little tricky, it’s a fantastic little trick that lets you use real RTL text without tedious backward typing.

Fake CloudApp with Dropbox and Quicksilver

2011-06-19T00:00:00-06:00

Recently a friend showed me CloudApp, a fantastic little app that sits in your menu bar and lets you share files almost instantly. With a quick keystroke you upload a file to their servers and get a public URL to share with anyone. Neato.

However, with a basic (free) CloudApp account you are limited in the size and number of uploads you can make each month. I’m also really picky about what goes in my menu bar, and I didn’t really like having an extra app running just for when I need to upload something.

My current workflow for uploading and sending files is to drag a file into my Dropbox public folder, navigate to it, right click it, and copy its public Dropbox URL. I like this because Dropbox is already running on my computer (no need for extra menu bar apps) and because I feel like I have more control over the fate of my file than with CloudApp (regardless of Dropbox’s recent legal issues). Plus there aren’t any arbitrary upload or storage space restrictions.

The one downside to this workflow is that it’s long and convoluted and lacks the magic instantaneousness of CloudApp. So I decided to make my own magical CloudApp-esque instant upload + public URL system with programs I’m already running: Dropbox and Quicksilver.

Here’s how to do it:

Download the “Post to Dropox” OS X service.
Unzip and open the Copy to Dropbox service in Automator.
Optional: If you don’t want to clutter your public folder with random files, you can change the “Copy Finder Items” action in Automator to point to a subfolder in your public folder. Mine is set to “~/Dropbox/Public/Uploaded,” so I can periodically go and clear out any out of date uploads, like one-off screenshots and the like.
In the Run Shell Script section, replace <DROPBOX-ID> with your Dropbox user ID (the one from your public URLs, not your e-mail address).
If you want to use the Move to Dropbox service as well (I use both), open it and type your user ID there as well.
Move both services to ~/Library/Services. You can now access them from the system wide services menu. Right click on any file, go to “Services” and easily move or copy any file to your public Dropbox folder and get the URL for that file placed automatically in your clipboard.

While these services eliminate most of the clicking and dragging I was doing before, they still require three clicks (select the file, right click to navigate to the services menu, click on the service). Luckily Quicksilver simplifies the process even more. You could probably use other Quicksilver-esque programs like Alfred and the like to get the universal service keyboard shortcuts.

Enable the “Services Menu Module” in Quicksliver’s plug-ins sheet in its preferences (⌘ + ,).
Go to Preferences and make sure “Enable advanced features” is checked.
Go to Catalog > Quicksilver and check “Proxy Objects”
Create a new trigger to pipe the current selection through each of the services. I use ⌃⌥⌘U for Copy to Dropbox and ⌃⌥⇧⌘U (essentially mashing down the whole corner of my keyboard :) ) for Move to Dropbox.
Select a file, press your new keyboard shortcut, and voila—the file is instantly copied to Dropbox and you have a URL ready to paste somewhere. Pure magic :)

One of the only differences between this and CloudApp is that there’s no response after you use the service. There’s nothing to let you know that it actually happened. However, you can easily add a sound or Growl notification to the Automator service if you want some sort of response.

So there it is. Magic CloudApp functionality using Dropbox and Quicksilver!

In Tahrir Square

2011-02-03T00:00:00-07:00

My awesome wife just wrote up a fantastic poem dedicated to the #Jan25 Tahrir protestors. This past week has been riveting and emotional for us, even though we live far from Egypt now. Cairo was our home for two years—we drank from the Nile (شربنا من النيل) and feel part of it. This horrific carnage is absolutely sickening.

Ta7ya Masr!

In Tahrir Square

In Tahrir Square the people fight
For freedom, truth, and human rights
We all should have; While here I sit
And watch the robins peck and flit
Not knowing what went on last night.

How many dead? We’ll never know.
The guns, the knives, the bombs aglow
Seek and will seek to thwart hearts knit
In Tahrir square.

Take up their quarrel with the foe:
To you from failing hands they fling
The torch; be yours to hold it high.
If ye break faith with those who die
They shall not sleep, when silence rings
In Tahrir Square.

Nancy Heiss
February 3, 2011

(With thanks to John McCrae)

Mona Prince on #jan25 Egyptian Protests

2011-01-27T00:00:00-07:00

Egyptian novelist and professor Mona Prince just sent me this personal report on her experience with the #jan25 protests. I’m reposting it here in its entirety.

a personal testimony on police brutality during protests: jan 26 2011 down town cairo–mona prince
round 6.30 pm down town cairo, i joined the demonstration on qasr el nil street along with some friends and other people whom i don’t know. the protesters were marching peacefully and politely and decently from one street to another evading the security forces who were some how at a loss at how to stop us, especially when we took to shawarbi street, we were at the back, and the police officers behind and front talking in their walki talki, we could hear them saying they don’t know how to besiege us. we kept on moving and more people were joining us from the streets and shops, young men and women. we were chanting the requests of egyptian people ” leave ” and ” egyptian people want the regime out “. we didn’t attack any body, or destroy any shops or cars… we were moving in the middle of streets without blocking the traffic, just slowing it down until we reached sherif street and we were heading towards 26 july street, then the security forces appeared from behind and front. we started running, and security police in civilian clothes started grabbing randomly many young men and women. i saw them grab and beat a young innocent man, pushed him to the ground and kept kicking. i protested against the beating up, and kept screaming at them to stop acting like animals, 4 or 5 huge giant men grabbed me from my hair and said ” well join him you bitch ” and slapped me on the face , beat me up, kicked me, and cornered me next to the young man and kept hitting me on my head, arm, shoulder, back , stepping on my head with their shoes until i bled from my mouth and could not speak any more on the ground. they kept their shoes on my head, kicked me every few seconds,calling me the dirtiest names ever the they dragged me all along the street to 26 july and threw me in one of those microbuses without number plates. they have also dragged the other young man who completely fainted and others too and threw us all in the microbus. and while pushing me inside they were trying to pull off my clothes and sexually harassed me,one grabbed my breasts , another held me my waste, and another grabbed my bottom. i tried to call friends discreetly, but they saw me, so they pull me out of the car and said ” get off ” but 3 of them were blocking the door, and they grabbed the mobile from me, then threw me to the asphalt road. despite the pain, i will go on protesting mona prince egyptian writer and university professor cairo january 27 2011

شهادة عن وحشية بلطجية الامن في المظاهرات- وسط القاهرة ٢٦ يناير ٢٠١١ مني برنس في حوالي السادسة و نصف مساء يوم الاربعاء ٢٦ يناير ٢٠١١، انضممت انا و أصدقاء و آخرين لا اعرفهم الى مسيرة سلمية احتجاجية في منطقة وسط المدينة، في شارع قصر النيل. و كان المتظاهرون يسيرون بكل ادب من شارع الي اخر في محاولة للهرب و تجنب قوات الامن المركزي الذين بدوا في حيرة و ارتباك حول كيفية محاصرتنا و تفريقنا، خاصة بعدما توجهنا الى شارع الشواربي. كنت انا و المجموعة التي معي في الخلف، و من امامنا و خلفنا ضباط الامن يتكلمون في اجهزتهم اللاسلكية و سمعناهم يقولون ما مفاده انهم لا يعرفون كيف يحاصروننا. استمررنا في السير بشجاعة و دون خوف، و اضم الينا الكثير من الشباب و البنات من الشوارع الجانبية و المحلات، و كنا نردد ” ارحل ” و ” الشعب يريد اسقاط النظام ” لم نعتدي على منشآت او سيارات و لم نهاجم احدا. كنا نسير و سط الشوارع دون أن نمنع مرور السيارات و ان كنا أبطأنا سيرها باتجاه شارع ٢٦ يوليو. ثم ظهرت قوات الامن المركزي من الامام و الخلف ر بدأت تجري نحونا بقوة، فجرينا متفرقين محاولين الاحتماء بالمحلات. و فجأة بدأ بلطجية الامن و وزارة الداخلية الذين يرتدون ملابس مدنية في القبض العشوائي علي الشباب و البنات و ضربهم بقوة. رأيتهم يقبضون على شاب كان يسير أمامي و لم يفعل اي شيء سوى الهتاف مثلنا، دفعوه بزخذيتهم و القوا به الي الأرض و انهالوا عليه ضربا و ركلا و لكما حتى اغمي عليه. اعترضت على ما يفعلون و صرخت فيهم الا يتصرفوا مثل الحيوانات و انهم ليسوا رجال و ليسوا بني آدمين، فاندفع نحوي ٤ او ٥ رجال ضخام الحجم و جروني من شعري، ضربوني علي و جهي، أخذوا يضربونني بعنف و يركلونني ثم القوا بي الى الآرض بجانب الشاب الذي فقد وعيه و هم يقولون ” طب يللا حصليه يا بنت القحبة “. فرد اخر ” مرة قحبة ” و استمروا في ضربي و ركلي بضعة دقائق و يدوسون علي رأسي و جسدي بأحذيتهم حتى سال الدم من فمي و هم يسبونني بأقذر السباب و الصفات، الى ان سكت و لم استطع الكلام. ثم جرجروني و هم ما يزالرن يركلونني و يسبونني الى شارع ٢٦ يوليو ثم القوا في واحد من تلك الميكروباصات التي لا تحمل لوحة ارقام. و فعلوا نفس الشيء مع شباب اخرين و القوا بهم الى داخل نفس السيارة، من بينهم الشاب الذي كنت ادافع عنه و فقد وعيه. و في اثناء محاولتهم حشري داخل السيارة، كانوا يتحرشون بي جنسيا، و يحاولون تعريتي واحد مسكني من صدري، و الحر من وسطي و شخص اخر مسكني من اسفل ظهري ثم دفعوني الى الداخل. كان هناك نحو ٥ او ٦ و ربما اكثر شباب داخل السيارة. حاولت الاتصال بأصدقاء بشكل هامس و بعيد عن اعين الامن لكنهم رأوومي. سحبوني من داخل الميكروباص و امروني بالنزول و هم يسبونني و في نفس الوقت يسدون باب السيارة ” بتكلمي مين يا بنت ….” و خطفوا مني التلفون المحمول بالقوة ثم شدوني خارج السيارة و القوا بي الي الاسفلت.

رغم الألم سوف استمر في الاحتجاج منى برنس كاتبة مصرية و استاذة بالجامعة القاهرة ٢٧ يناير ٢٠١١

iOS 4, Multitasking, and Battery Life

2010-09-24T00:00:00-06:00

I recently got a brand new 4G iPod Touch for my initial foray into the amazing world of iOS. I’ve been sitting on the sidelines far too long and I’m so excited to finally have an almost-iPhone.

However, I ran into my first hitch this morning and my Google skills have proven useless to solve it. I’ve been using the iPod as an alarm clock for the past few days since its UI is superior to the ancient digital clock in our bedroom :). When I went to bed last night the battery was probably at 80% capacity. Apple touts that the new iPod has 40 hours of battery life when playing music—quite impressive. I had put the iPod in airplane mode, turning off the WiFi so that I wouldn’t get any mail or Facebook notifications. With no internet, no music, and no apps (supposedly) running, the iPod’s battery theoretically should have lasted for months—or even years :)

However, I didn’t wake up to the iPod’s alarm this morning. I woke up to Rachel pulling on my arm and saying “Wake up, Dad! I want cereal!” 1.5 hours later than when I was planning to wake up.

The iPod’s battery had died in less than 7 hours.

I’m assuming this is linked to the new multitasking system in iOS 4, where apps that aren’t running are put into memory so you can open them where you left off. Sure enough, I had like 25 apps in the multitasking tray, which I quickly cleared out after plugging in the depleted iPod.

So, iOS users out there: what experience have you had with multitasking and battery life? Is there an easy way to “officially” quit an application rather than send it to the tray? What should I do avoid something like this again?

Queen Rania at AUC

2010-02-28T00:00:00-07:00

This morning AUC’s John D. Gerhart Center for Civic Engagement hosted a lecture by Queen Rania al-Abdullah of Jordan. Queen Rania is famous for being intensely involved in the public sphere and, according to the venerable Wikipedia, is considered one of the world’s most powerful women. She’s involved with a ton of foundations and NGOs that cover a wide range of goals, from advocating for improvement in girls’ education and employment, promoting dialogue between the US and the Arab world, and calling for what became the buzzword of today’s lecture: civic engagement. She uses technology to promote her agenda of social improvement and is active on YouTube, Twitter, and Facebook. Basically a beautiful, famous, powerful world leader.

Queen Rania focused primarily on the need for native grassroots movements in the Arab world, stating that civic engagement is an essential element of societal reform—changes so desperately needed in this region, where poverty and corruption are rampant. She stated that it is the responsibility of all citizens to take an interest in developing and improving their own country. Calling on Arabs to “look up” and think of others, she cited the example of one of AUC’s homegrown NGOs, Alashanek Ya Balady, which is heavily involved in promoting sustainable development in Egypt’s poorest neighborhoods. The efforts of AYB and dozens of other organizations constitute a laudable homegrown effort to reform Egyptian society, and they have had lots of success so far.

The queen then went on to decry the current state of civic laziness in the Arab world, saying that many Arabs confuse the responsibilities of “citizenship” with “sitizenship,” opting to sit still, sit back, and complain rather than take their own initiative to change society. This is not because Arabs are disconnected, emotionless people—far from it. “We’re all passionate about food, family, football, and Filistin,” she said, but those passions need to be refocused on bringing about change instead of complacently complaining about the backwardness of Arab governments and societies.

She developed this theme by responding to a series of preselected questions that followed her main motivational speech, declaring that universities and academia are responsible for remaining engaged with society and not remain isolated in academia. Too often the educated class remains cynically aloof from the rest of the world and fails to contribute real change or instill the culture of civic engagement in their pupils. She also noted the role of Islam in developing this culture of engagement, saying that Islam inherently advocates civic reform and grassroots movements. She called on the audience to “recapture those compassionate values of Islam” that have been hijacked by Western media and use the power of faith to help improve society.

Essentially, she concluded, if we want to see any reform and improvement in the region, the onus is on citizens. Change in the Middle East cannot come from governments, since they are inherently inefficient. Real change can only come about with a change in attitude and an increase in engagement. Regular citizens are the driving factor for this change.

In the end it was a pretty motivating and inspiring speech, and I agree with most of what she said. Real, lasting change in the Middle East will need to come from some type of bottom-up popular movement, especially when the reforms so desperately needed reduce the power of the ruling class. However, repressive governments throughout the Arab world (including Rania’s Jordan) severely limit what progressive movements can do, and Middle Eastern dictatorships last for decades.

For example, Egypt’s current president, Hosni Mubarak, has been in power for almost 30 years. He ran the country uncontestedly until 2005, when, after pressure from George W. Bush, Mubarak amended the constitution to allow candidates from parties other than the ruling National Democratic Party (NDP) to run in presidential elections. The first “free and fair” elections were held that year. Despite a somewhat large grassroots political opposition movement, Kifaya, Mubarak won handily and his opponent, Ayman Nour, was arrested and sentenced to prison for five years. Today Kifaya, a native grassroots movement, is detoothed and powerless. Good thing civic engagement worked…

The next Egyptian presidential elections are slated for 2011, but since Mubarak is getting on in years (he’s 81!), he’s probably not going to run. Instead, it seems that his son, Gamal Mubarak, will inherit his father’s place. Sure, he’ll go through the farce of the election process, but the NDP politicos and thugs will pretty much guarantee his victory.

Understandably, there is a growing anti-Gamal movement in Egypt, and it’s even homegrown and grassrooty. It got a huge boost last week when Mohamed ElBaradei, former head of the IAEA and winner of a Nobel Prize, returned to Cairo after a decades-long pseudo-exile. He has announced that he would consider running for president in 2011 if circumstances allowed for it. He already has a large base of support and even an official-ish campaign website. American media is picking up on him, too. He has a lot of international clout and could present a real threat to the Mubarak dynasty. He might have the potential of becoming the Egyptian Obama—“Yes we can!”

But (and this is a pretty big but…), he can’t even legally run—he’s not constitutionally permitted to run for president. For him to become president, Mubarak would first have to amend the constitution to open the field for more opposition parities. Then ElBaradei would actually have to win in elections run by the NDP. Yeah. Good luck there.

While the possibility of an ElBaradei run has Mubarak somewhat scared, the cards are really stacked against him. This growing popular movement faces the impossible task of forcing a constitutional amendment. It’s a native movement, just like Queen Rania wants, but it is severely limited. Egyptians remain hopeful, but sadly, Gamal will most likely take over in 2011.

While Jordan, as a monarchy, doesn’t face these issues of presidential elections, it has its own slew of problems with citizenship. Although Rania herself is a Palestinian, the Jordanian government regularly withdraws Jordanian nationality and rights from Palestinian refugees. How can native, grassroots Jordanian organizations reform when the government makes them stateless?

In her speech Rania exhorted AUCians: “Don’t wait for governments to change and reform. You must be the change you want to see.” Um. She’s in the Jordanian government. She’s the queen! She surely has some influence on the Jordanian political scene. Can’t she help the government change so that these popular groups can actually do something? Arab governments tolerate, even embrace, NGOs like AYB or the queen’s water projects in Jordan—they love this type of civic engagement. But heaven forbid these civically engaged movements threaten their despotic power, though.

Can real governmental and societal change happen in the Middle East solely through “civic engagement.” No way. Arab governments must make changes if any popular movements want to make substantial change.

So, while Queen Rania’s speech was inspiring and motivational on the surface, it was full of kalaam fadi—empty words.

flashbakectl released

2009-09-23T00:00:00-06:00

Adding to my apparent series of Flashbake addons, I’ve just released flashbakectl.

Normally to run Flashbake consistently you need to set up a cron job. While OS X is built on Unix and has cron, Apple recommends using launchd and property list (plist) files to run system agents and daemons. flashbakectl is a handy little script that loads and unloads a plist for you.

Before working on your project, run flashbakectl -l to load the plist and start the daemon, which will commit your unsaved changes every 15 minutes (or whatever you set it to). When you’re done for the day, run flashbakectl -u to stop the daemon, saving your computer from unnecessarily running Flashbake ad infinitum.

flashbakectl only works on Mac OS X. You can get it at GitHub. Enjoy!

iTunes plugin for Flashbake

2009-08-18T00:00:00-06:00

Flashbake is a fantastic script for nerdy writers (like me) that periodically commits changes to a Git repository and can optionally append various metadata to the commit message, allowing you to annotate the entire creative process.

Flashbake includes several plugins for adding recent tweets, weather, the current time zone, and other random information. There’s even a plugin for the Banshee music player for Linux. There’s nothing for iTunes, however, which is unfortunate since I’m always listening to something when I write or code.

So I hacked together a little plugin for Flashbake that uses AppleScript to get the current track information from iTunes and add it to the commit message. It’s admittedly a “frankenscript” and only works on Mac OS X (since it relies on AppleScript), but it works great.

You can get it at GitHub. Enjoy!

Using Google Voice and Gizmo Project Together

2009-08-01T00:00:00-06:00

See update below (skip to update)

Google Voice, the Google-ized incarnation of GrandCentral, is a fantastic service that aims to become your virtual phone switchboard. It gives you a free phone number that can receive regular phone calls and route them to any other actual phones you have connected to your account. Powerful stuff.

Unfortunately, though, its forwarding abilities are limited to US phones. Using some VoIP magic, though, you can create a semblance of international forwarding and get free (or nearly free) phone calls to the US while abroad. If you’re not in a foreign country, you can harness the same VoIP magic to get a nearly free phone service.

The Gizmo Project–background

Gizmo is normally a VoIP provider that lets you make free (or super cheap–something like $0.019 a minute) phone calls. When you sign up for an account you get a special phone number in the 747 area code as your VoIP/Gizmo username. While any phone on the Gizmo network can call your 747 number for free, regular phones can’t connect to it.

Gizmo offers a Call In service that lets you buy a phone number in most US area codes (or one of dozens of countries), which then lets you receive phone calls from standard phones. Call In numbers start at $35 a year (or $12 a year for three months), but prices can be higher depending on demand.

Gizmo touts itself primarily as software–it provides a “soft phone” program that you run on your computer. As long as the program is open you can make and receive phone calls (much like an IM program or Skype) using a microphone and your computer’s speakers or headphones.

Gizmo without a computer

It’s impractical to keep your computer on all the time and it can be awkward to use your computer as a phone. You can get around this limitation by buying an ATA adapter–a little box that plugs into your network with the sole purpose of running phone services. It’s essentially a hardware version of the Gizmo soft phone.

Fancy corporate VoIP phones (like the ubiquitous Cisco ones) have ATAs built in (kind of. The real ATA is somewhere on the network letting these computer-phones connect to it). You don’t need a fancy VoIP phone, though. Standard consumer ATAs let you plug regular phones directly into the adapter.

After configuring the ATA with your Gizmo information you can make and receive calls using your Call In number. Rather than pay $40+ a month for regular phone service, you can have a fully featured phone that only costs $35 a year plus <$0.02 a minute.

Before Google Voice

For the past three years we’ve been using Gizmo as our full-time phone. We bought a Linksys/Sipura ATA (our model, the SPA1000 is no longer manufactured), got a $10 phone from Target, and bought a Utah county 801 Call In number. The phone worked perfectly. I could plug the ATA in to any internet connection and get cheap/free phone service.

Our system even works (mostly) in Egypt. I have two phones on my desk: the $10 Target phone plugged into the ATA (which is plugged into the router) and a 20 EGP neon pink phone (plugged into our Egyptian phone line). Family, friends, and unsuspecting telemarketers can reach us at our Utah number and pay only what it costs them to call an 801 number.

Our current phone set up

The only problem with the system our system in Egypt is a bizarre limitation with Egyptian (or at least Link.net’s) internet infrastructure. We don’t have any bandwidth issues when someone calls us, but when we call out, the connection drops within the first ten seconds of the call 90% of the time. To get around this, we used some SkypeOut credits–we’d call someone in the States with Skype (using my computer), tell them to call us on our Utah number, hang up, and wait for their call.

This worked when we called actual people, but doesn’t work when calling banks, insurance companies, airline companies, or anything else with a phone tree–phone trees can’t call you back. SkypeOut works for those, but it’s more expensive than Gizmo.

Google Voice changes all this.

Enter Gizmo Voice

Google and Gizmo have joined up to let you hook Google Voice directly into your Gizmo account. Rather than buy a Gizmo Call In number, I can use my free Google Voice number with my Gizmo Account. After following Gizmo’s instructions on connecting the two accounts, now when people call my Google Voice number, the call is routed to the normally inaccessible 747 Gizmo number, which is already associated with my ATA box.

This means I can stop paying $35 a year for my Call In number. The only thing I pay for is the phone use itself. Gizmo just changed their phone rates for users using Google Voice–apparently all calls under three minutes are free, while longer phone calls follow their normal low rates.

Additionally, now that I can have my Gizmo phone connected to my Google Voice account, my bizarre issue with calling out on Egyptian internet can be solved. In order to call people and have your Google Voice phone number appear on their caller IDs, you need to use Google Voice as an intermediary. You type in the number you want to call on their website and they’ll call one of your linked phones. When you pick up, your phone will start dialing the outbound number. Since Google calls my Gizmo phone now to make outbound calls, Link.net considers it an inbound call and it doesn’t get cut off.

So now, Gizmo combined with Google Voice gives me free short calls and cheap long calls to the US and a free US number that can replace my Gizmo Call In number. Everything works both in the States and internationally. It’s a nearly perfect system.

Still untested Now tested…

In theory, since I have my Gizmo and GV accounts linked, my Google Voice number should show up on the recipient’s caller ID when I call out with my US phone, circumventing the need to use the Google Voice web site as the middleman. ~~I can't test it, though, since my internet connection won't let me make any outbound calls with my ATA. I'll keep trying over the next week, since I can make like 10–15 second calls 10% of the time.~~

It works! My Google Voice number shows up, just like it should…

Update

Google has acquired Gizmo5, which hopefully means that the link between Gizmo and GV will be more permanent and more official. Awesome.

Alexandria Train Crash

2009-07-30T00:00:00-06:00

See update below (skip to update)

After almost a year of being in Egypt, we finally decided to go up to Alexandria today. We took the 9:00 AM train from Cairo and the ride went smooth until we were just outside the final Mahatat Misr station, where we were delayed for over an hour until pulling up to a platform.

Immediately after we got off the train we saw the reason for the delay. A train had rammed into the station, apparently at full speed, derailing the first three cars—the engine, the generator, and the first actual cabin. Part of the station itself was damaged, as was, ironically enough, a parked fire truck.

Click to enlarge

The train crashed at around 8 AM and was empty. By 12:30 (when we got there), Egyptian Railway repair crews were lifting the damaged cabin off the tracks.

By 6:00 PM, when we arrived back at the station to return to Cairo, the repairmen were working on lifting the engine and had apparently already extricated the generator car.

Since two platforms were out of commission, trains were shuffled around all day, causing major delays. Our train ride back to Cairo took over four hours instead of the usual two.

As far as I know, nobody was killed as the train was empty. I don’t know about injuries, since I arrived on the scene several hours after the actual accident.

Needless to say, it was exciting—in a macabre sort of way. I’ve read so much about Egyptian train crashes—I got to see the aftermath of one. :)

~~I'll post the full set of pictures tomorrow on Flickr.~~

Update

I posted our photos of the accident at Flickr. They are licensed under a Creative Commons Attribution license, so you can use them however you want.

Nancy has also blogged about the accident at our family blog.

Additionally, the online Egyptian news site Youm7 has posted an update (in Arabic). According to their report, the accident was caused by a brake failure as the train travelled from the nearby Sidi Gaber train station. It was indeed empty, fortunately, and there were no fatalities. Two of the crew members were injured in addition to a vendor—since there are snack booths at the end of each platform, he must have gotten smashed. There are conflicting statements as to the cause of the accident. Some claim neglect and disrepair; others sabotage. I’m leaning towards neglect—those trains are ancient.

Installing pdftk-php

2009-07-29T00:00:00-06:00

Upon popular request, I’ve decided to update the original tutorial for populating a LiveCycle PDF with PHP to apply to the new release of pdftk-php. The installation instructions should be mostly clear in the readme and in the inline comments in the example included with the script; this post is merely supplemental.

Basic usage

Initial set up

Download the most recent version of pdftk-php from GitHub and download and install pdftk on your server.

Unzip the download from GitHub and place the folder on your server. I’ve placed mine in a folder called pdftk-php.

Create a MySQL user and database and run the SQL found in /example/database.sql in a MySQL client (like phpMyAdmin) to create the sample database.

Modify the information in example/_dbConfig.php so that the application can connect to your database.

$host = "localhost"; 
$username = "pdftk-user"; 
$password = "supersecure"; 
$db_name = "pdftk-php";

Browse to the example site (in my case, http://localhost/pdftk-php/example/index.php) and add some entries to populate the database a little.

Set up the script

Open pdftk-php.php and insert the full path to your working pdftk installation at the beginning part of the passthru() command near line 71. Here are some examples for different scenarios on server platforms:

// On a typical Unix-based installation
passthru("/usr/local/bin/pdftk ...");

// On Windows, with an absolute path
passthru("c:\pdftk\pdftk.exe ...");

// On Windows, with a relative path (useful if you place pdftk.exe in the server folder structure)
passthru("../pdftk.exe ...");

If you’re on a Unix-based server and don’t know where pdftk is, type one of the following commands, which should result in the absolute path to the program:

which pdftk
# or
whereis pdftk

In example/download.php verify that the path to the required pdftk-php.php is correct, near line 18. In the example, pdftk-php.php is located a directory below the example directory. If you like to store your included files elsewhere, make sure that you modify the require() path here.

pdftk-php.php needs to be able to write to a temporary directory on your server in order to create a temporary FDF file. This directory is specified near line 58, with the tempnam() function.

If you are on a Windows server you should already be able to write to pretty much any directory (I think… I’ve never worked with IIS permissions), so you should be good to go. If you are on a Unix-based server you’ll need to be more explicit with directory permissions. To make things easier, create a temporary folder on your server and give it write permissions:

cd pdftk-php
mkdir tmp
chmod 777 tmp

Then set the path in tempnam() to the new temporary folder.

// If at the same level as download.php
$fdf_fn = tempnam("tmp", "fdf");

// If one directory behind download.php
$fdf_fn = tempnam("../tmp", "fdf");

// You can also use an absolute path
$fdf_fn = tempnam("/Library/WebServer/www/pdftk-php/tmp", "fdf");

Set up the PDF

Create a fillable form in either Acrobat Professional or LiveCycle Designer, or use the included example PDF form. Give each field a unique and significant name so that you can work with the form more easily later on. You can modify field attributes by double clicking on the field using the Forms toolbar in Acrobat; in LiveCycle, use the Object panel.

If you are using LiveCycle, you’ll need to save the final PDF as a static form compatible with Acrobat 7. pdftk doesn’t work with dynamic forms or PDFs from later versions of Acrobat.

Connect PDF to script

example/download.php connects to your database, retrieves a row based on a passed GET variable, saves the data from the fetched row into variables, finally calling pdftk-php.php, which does the heavy lifting of creating an FDF file and injecting it into the PDF.

Starting at around line 30 the script assigns the fetched values to variables. Each of those retrieved variables needs to be paired with a form field in your PDF (near line 39). In a basic Acrobat form this is simple:

$fdf_data_strings= array('firstname' => $pdf_firstname,  'lastname' => $pdf_lastname, 'email' => $pdf_email);

LiveCycle tends to complicate the form names slightly. You can use pdftk from the command line to retrieve the official form field names. Run this command from the directory containing your PDF file:

pdftk form.pdf dump_data_fields > form-fields.txt

When you open the resultant .txt file you should see a report of all the fields

...
---
FieldType: Text
FieldName: form1[0].#subform[0].firstname[0]
FieldNameAlt: First name&#9;
FieldFlags: 0
FieldJustification: Left
---
...

Use those long, hairy FieldNames in the $fdf_data_strings array, like so:

$fdf_data_strings= array('form1[0].#subform[0].#area[0].FirstName[0]' => $pdf_firstname, 'form1[0].#subform[0].#area[0].LastName[0]' => $pdf_lastname, 'form1[0].#subform[0].#area[0].EMail[0]' => $pdf_email, );

Finally, check the values of $pdf_filename and $pdf_original near lines 62 and 65.

Go to http://localhost/pdftk-php/example/view.php and click on the download links for one of entries. You should be prompted to download a PDF file, dynamically generated using pdftk-php.php. Success!

Advanced customization

Using checkboxes or radio buttons

$fdf_data_strings works great for text fields, but can’t handle radio buttons or check boxes. For that you’ll need to use the $fdf_data_names array near line 49.

NB: The logic for manipulating the form data in PHP and MySQL might be a little convoluted and could easily be optimized, but it works for clear demonstration purposes.

To demonstrate this how to do this, we’ll add a checkbox to our form and extend the database. Run this query in a MySQL client to add a couple columns to our table:

ALTER TABLE `users` ADD `option1` TINYINT( 1 ) NOT NULL, ADD `option2` TINYINT( 1 ) NOT NULL ;

Open /example/example.pdf in Acrobat Professional and add two checkbox fields named option1 and option2.

We need to modify our web form and the table that displays the data, just to make sure everything is getting saved to the database correctly.

First, make a couple changes to example/index.php After the section near lines 104–107, add

<p>
	<label for="option1">Option 1</label>
	<input type="checkbox" name="option1" value="1" id="option1" />
</p>
<p>
	<label for="option2">Option 2</label>
	<input type="checkbox" name="option2" value="1" id="option2" />
</p>

Then, up near the top of example/index.php after line 34, add this:

if ($_POST['option1'] == 1) {
	$option1 = 1;
} else {
	$option1 = 0;
}

if ($_POST['option2'] == 1) {
	$option2 = 1;
} else {
	$option2 = 0;
}

This checks the value of the submitted checkboxes and sets the $optionx variables to either 1 or 0, which fit into the TINYINT columns in our table. You could use actual text as well and set the columns to VARCHAR.

Change the SQL query from

$sql = "INSERT INTO users (firstname, lastname, email) VALUES ('$firstname', '$lastname', '$email')";

$sql = "INSERT INTO users (firstname, lastname, email, option1, option2) VALUES ('$firstname', '$lastname', '$email', '$option1', '$option2')";

Go ahead and insert some dummy submissions with boxes checked and unchecked to make sure everything is working.

Optionally we need to modify example/view.php to show the stored values. Add the following table header cells after line 29:

<!-- Already here --><th>E-mail Address</th>
   <th>Option 1</th>
   <th>Option 2</th>
<!-- Already here --><th>Download PDF</th>

In the while loop a few lines later, add this code:

<!-- Already here --><td><?php echo $user["lastname"]; ?></td>
<td><?php echo ($user['option1'] == 1) ? "Yes" : "No"; ?></td>
<td><?php echo ($user['option2'] == 1) ? "Yes" : "No"; ?></td>
<!-- Already here --><td><?php echo $user["email"]; ?></td>

This is just PHP ternary notation, which essentially says that if the value of optionx is equal to one, echo “Yes,” otherwise, echo “No”.

Finally we need to modify example/download.php to handle our checkboxes. Like I said above, the $fdf_data_names variable handles checkbox and radio button data. In PDF forms, the two allowed values for checkboxes are “Yes” and “Off” (not really opposites, but oh well), so you’ll need to set variables accordingly. Replace $fdf_data_names = array(); near line 49, with this, which checks the values of optionx and sets $pdf_optionx to either “Yes” or “Off” and then defines the $fdf_data_names array appropriately:

if ($data['option1'] == 1) {
	$pdf_option1 = "Yes";
} else {
	$pdf_option1 = "Off";
}

if ($data['option2'] == 1) {
	$pdf_option2 = "Yes";
} else {
	$pdf_option2 = "Off";
}

$fdf_data_names = array('option1' => $pdf_option1, 'option2' => $pdf_option2);

And that should do it! Visit http://localhost/pdftk-php/example/view.php and download one of the forms. The checkboxes should populate perfectly.

Other types of form fields

Combo boxes and radio buttons act similarly to checkboxes. If you run the dump_data_fields command with pdftk again on a form with these more advanced options, you’ll see a few differences in the results.

FieldType: Text
FieldName: email
FieldFlags: 0
FieldJustification: Left
---
FieldType: Button
FieldName: option1
FieldFlags: 0
FieldValue: Yes
FieldJustification: Left
FieldStateOption: Off
FieldStateOption: Yes
---
FieldType: Choice
FieldName: favoriteColor
FieldFlags: 131072
FieldValue: blue
FieldValueDefault: red
FieldJustification: Left

You can see the “Yes” vs. “Off” values in our checkbox (called “Button” in PDF lingo). Drop down lists (“Choice” in PDF-speak) have multiple values, specified by you when you create the field.

Radio buttons are hybrids. They are considered “Buttons,” like checkboxes, but can have custom values, like drop down lists.

Your final PHP script will need to take these different values into account and assign the correct values in the $fdf_data_names array.

Conclusion

You can do a ton with the pdftk-php.php class once you get it set up initially and get past the slight learning curve. If you have any questions, feel free to ask in the comments. If you find any problems, comment here or open an issue at the GitHub project page. Additionally, you can fork the project and contribute.

Good luck!

On narrowing and redefining research

2009-07-28T00:00:00-06:00

I’ve been in Egypt for almost a year now, studiously working towards my MA in Middle East Studies. The supposed capstone of my time here at AUC—my thesis—now looms ahead somewhat menacingly. I get to spend the next several months researching and writing what will end up being my largest research project to date. It’ll also set the foundation for my (hopefully) future PhD plans.

There’s only one problem: I don’t quite know what I’m writing it on.

At BYU I double majored in Middle East Studies/Arabic (MESA) and Italian—an odd mix of modern history, political science, and renaissance poetry. Most of my research focused on finding literary and historic connections between the Middle East and Italy. I looked at the role of Sufism in the birth of Catholic mysticism, especially with Jacopone da Todi and St. Francis of Assisi. I looked at the Young Ottomans and their reliance on Mazzini’s Giovine Italia ideology. I even presented a paper at a conference about the role of Mohammed in Inferno XXVIII in Dante’s Divine Comedy, connecting it to proto-orientalism.

Fun times :)

At the same time, though, there was an inherent conflict in my research interests. I wrote my MESA capstone paper on the media coverage of the 2006 Israel/Lebanon war, where no Italian literature was involved :). I love media—I’m obsessed with the news, the internet, blogs, Twitter; anything shiny, new, and exciting.

I started my MA with the assumption that I’d have to choose one of these tracks—history or media. I dove headlong into the history track, writing a huge literature review on the history of the Italians in Egypt at the beginning of the 20th century. Large Italian communities in Cairo, Alexandria, and Ismailiyya sprang up after Napoleon’s 1798 invasion. Thousands of Italians were born and raised abroad in these cities, and many took part in the Egyptian nationalist movement, considering themselves more Egyptian than Italian, thus earning themselves the nickname mutamasirun (those who try to be Egyptian).

I was excited about this research until a new shiny thing took my attention away—Twitter. I started using Twitter more or less full time in January, which then introduced me to the large Egyptian Twitter community. Many of the Egyptian twitterers are also activist bloggers who have been arrested multiple times. I began to see the power and potential of the internet in political reform and change in the Middle East and switched research gears to focus on the Middle Eastern blogosphere. It was fascinating stuff, but I felt like I had turned my back on history :) Further complicating things, I had an amazing history seminar last semester that resulted in some awesomely fun archival research. I had a blast writing the paper. The primeval dichotomy of history vs. media reared its ugly head again.

Since I’m nearing the end of my MA, I’m looking at different PhD programs to go to once I’m done here. There are lots that look at bloggers and politics—even ones that look at bloggers and politics and the Middle East—but none do it with a historical approach (fancy that… blogs have been around for something like four years and it’s not history yet :) ). I sent out dozens of e-mails to different professors asking about graduate programs and my potential research. Most responses were confused; history != new media.

However, one professor at Cornell responded positively. He studies media in modern Egyptian history, specifically in the time period of my Italian mutamasirun.

I think I may have found a way to bridge the history/media studies gap. Now I just need to figure out exactly how to do it.

Awesome. :)

So, sorry bloggers—I’ll keep following you as a tangential fascination, but my heart lies in history.

Import a Blogger Blog to InDesign with Perl

2009-07-19T00:00:00-06:00

Our family has a fairly sizable blog that we (actually, mostly my wife, Nancy) have kept updated for several years. Since it contains so much family history we wanted an easy way to preserve it in print form, just in case Blogger gets the boot from Google some day (not that that will ever really happen…).

Since we’re both hobbyist graphic designers—I taught a couple print layout and design classes as an undergrad at BYU and have made several books at Lulu.com—we decided to layout and print each year of our blog, to keep for posterity.

A couple years ago Nancy attempted this with our smaller Jordan blog for a print publishing class she took at BYU. We spent the bulk of our time manually copying and pasting each post and the subsequent comments into a huge Word document. She then ran a long series of find/replaces to clean up the messy, inconsistent typography, and then finally placed it into Quark (that evil program). Through a series of unfortunate events, Quark crashed repeatedly and corrupted her file multiple times—she was lucky to get her first draft turned in for her final project (she got an A, though. Phew!).

I knew there had to be a faster, more efficient way to wrangle all the blog text, but this was back in 2006, before Blogger had an open API or options to backup a blog. Primitive, dark days indeed :).

However, last year, Blogger introduced a fantastic new option—the ability to backup and export your entire blog, comments and all. Blogger spits out an Atom-formatted XML file that you can use to recreate your blog later on (or possibly import onto other platforms, like WordPress, I think). This was the key to simplifying the daunting task of collecting the text for our blog books. All we needed was a way to mangle the text in the XML file to create an InDesign-ready file.

So, I whipped up a semi-complicated Perl script that can parse an Atom-formatted XML file from Blogger and create a text file using InDesign Tagged Text to preapply paragraph and character styles. It also cleans up the typographic elements of the text, adding em and en dashes, removing empty paragraphs, etc. Additionally, it can add hidden index entries for each tag, essentially creating a barebones index for your book. And it only takes 10ish seconds to run on a large blog. It’s not perfect and could stand some good optimization, but it works.

Additionally, since InDesign tagged text works with, well, text, it won’t place your images for you. Instead it will insert the location of the image (the src=whatever.jpg of the img tags) in between curly braces { }. You’ll then need to manually place all the images later, deleting the braced text.

In the future, the script could be changed to output XML, which does let you include pictures, but you’d have to have all your images on your hard drive already. The script could go and download all the linked images, but it’s not really a good idea to place low resolution, web-optimized images in a print document. In our case we have high-res copies of all the pictures on the blog stored on an external hard drive, so we just have to go and find and place the images we want. It takes more time, but it makes better quality documents in the end.

Also, links are preserved as footnotes—all href="whatever.html"s show up as the footnote text.

How to use the script

First, download the script and its supporting files from Github. If you’re using Mac OS or Linux, make sure the main script file, format_for_id.pl is executable—type chmod +x format_for_id.pl at the terminal.

Next, make sure you have Perl installed on your system. If you are using Linux or Mac OS X, you’re good to go. If you’re using Windows, download and install Strawberry Perl for Windows. You can also use ActivePerl, but installing modules is a little more difficult.

The script uses several additional CPAN modules that you’ll need to install. You’ll need to use the CPAN shell to do so.

On Windows with Strawberry Perl: open the packaged CPAN client in the Start Menu folder
On Windows with ActivePerl: Good luck. There is a large repository of specially compiled CPAN modules for ActiveState, and reportedly there is a kind of CPAN shell, but I haven’t gotten either to work too well. Stick with Strawberry Perl. It’s better :)
On Mac OS X: type perl -MCPAN -e shell at a terminal window
On Linux: type sudo cpan at a terminal window

(If it’s your first time running the CPAN shell you’ll be asked to configure the installation environment. Choose the option to automatically configure everything.)

Once everything is set up and you see the cpan> shell prompt, type install Package::Name (eg. install Date::Format) for each of the dependent CPAN packages listed at the beginning of format_for_id.pl.

Log in to your Blogger Dashboard and export your blog as an XML file by going to Settings > Basic > Export blog. Place the XML file in the script folder.

Open config.cfg with a text editor and change the settings as needed. Set the input file to your newly downloaded XML file, choose the year you want to extract, set an output file, and set the file header, either <UNICODE-MAC> or <UNICODE-WIN>, depending on what platform you use InDesign on.

For now, leave all the style tags as they are so you can place the text into the example InDesign file and see how everything works. You can change them later and rerun the script

Finally, using the terminal or command prompt, navigate to the folder with the script and and run it by typing perl format_for_id.pl. If everything goes well you should have an output file at the location you specified, full of InDesign tags.

Open up Example.indd in InDesign CS3 or above and place the generated text file. All the text should come in perfectly with all the needed paragraph and character styles applied. Bravo!

Advanced usage

Obviously you’ll want to make some changes to the format of the output text. You might not want the post URL right after the tag—you might want it at the end, or not want it at all. With a little knowledge of Perl, you can edit the main script directly, mostly the combineSortClean() sub near the end of the script, to change the order of the output elements.

You can also disable tag indexing and allow the tags to be output with a paragraph style. Just comment and uncomment the appropriate sections in the code. The same goes for the author-specific character styles—comment and uncomment the needed lines in the script.

You can rename the styles and use your own—just make sure the styles exist in your InDesign document before you place the output file. InDesign will throw away any tags that don’t already exist in the document.

I made the script for our specific blog, so it doesn’t take every possible paragraph or character style into account. If you want additional functionality, you’ll have to add it. Feel free to fork the project off of GitHub and add to/improve it. That’s why it’s open source :)

If you have any questions, ask in the comments. Report any issues at the project GitHub page. I’ll try to respond quickly—I generally do, as evidenced by my pdftk-php project :)

Good luck!

pdftk-php Officially Released

2009-06-19T00:00:00-06:00

Wow. It’s been almost two years since I wrote a little tutorial on how to use LiveCycle, PHP, and MySQL together to make a web application that served dynamic PDF forms. Since then it has become the number one page on this site. I still get a substantial number of comments a week here on the blog and via e-mail—many of those comments are stuck in my inbox, sent to me before I rebuilt my site on WordPress and enabled commenting.

Unfortunately, though, I wrote that tutorial as my first foray into the world of PHP/MySQL web development and had little idea of what I was really doing. Since then, however, I’ve done a fair amount of real-world web design and development, and even implemented this pdftk form system into a live, public-facing application.

In the interim, I’ve refined my system and released it as an open source PHP class, named pdftk-php. The project is hosted at GitHub, a brilliant hosting service for collaborative projects, which uses Git, the best version control software I’ve ever used. Anyone can check out, or clone, the project, make any edits to the core set of classes, and merge those with the main project branch—it is now a true community project. If you don’t want to contribute, you can still download it from GitHub as a .zip or .tar file.

Included in the project is a (hopefully) extensively documented example application that you can set up on your own server. You can try a live working example at pdftk-php.andrewheiss.com.

I’m working on writing a new step-by-step tutorial on how to set everything up, akin to the old one. In theory, the new pdftk-php should work with PDFs created in any program, not just LiveCycle. In the meantime, download pdftk-php, try it out, and report any bugs here on the blog or directly at GitHub (where I hope to keep everything related to pdftk-php from now on). Fork the project and contribute if you feel like it, too!

Thanks and good luck!

A Tale of Three Taxis

2009-05-14T00:00:00-06:00

Traffic in Cairo is horrible, especially when all the taxi drivers, the ubiquitous life-blood of the Egyptian streets, have a deathwish for you.

Read about my recent near misses at my post at our family blog, Heissatopia.

Google Profile Business Cards

2009-05-01T00:00:00-06:00

Look what I just ordered, for free:

Awesome :)

Google is giving packs of 25 of these away for free to promote their new Google Profiles service. I think the profile idea is great—it makes finding people online so much easier and gives you more control over what Google says about you.

Too bad I won’t get these until August when my mother-in-law comes to visit.

Typing transliterated Arabic quickly

2009-04-26T00:00:00-06:00

Since Arabic doesn’t use the Latin alphabet, and lots of the letters don’t have Latin equivalents (خ, ع, ق, ط, for example), transliteration is necessary to show Arabic words and sounds in English writing. There is an easy way to type transliterated Arabic quickly, though, using macros to locate hidden Unicode characters used by many of the standard transliteration systems.

Unfortunately, there is no universally standard system for transliteration, and most systems use letters that aren’t found on normal keyboards. One of the rising systems in the Middle East, nicknamed Franco Arab in Egypt, is my least favorite. It only uses standard English letters, meaning it’s useful for texting, e-mailing, and other things where it’s difficult to write in real Arabic script. Biggest problem: it’s ugly and hard to read.

For example, the name Great Britain (بريطانيا العظمى) uses several non-Latin letters. Written in Franco it looks like this: bri6ania al3o’6ma. For readers unfamiliar with Arabic (or even those who are, like me), it’s always hard to remember what the random uppercase letters and numbers mean.

Fortunately, there are better systems. Here’s Great Britain written using the IJMES (International Journal of Middle East Studies) system, also used in the Encyclopedia of Islam: Brīṭānīyā al-ʿuẓmá. Much easier to read.

Since the nonstandard Latin letters use Unicode glyphs, you need to use a font that has a full set of Unicode glyphs, like Times, Arial, Helvetica, and other standard fonts. You also have to hunt down all the special characters either in Word’s Insert Special Character dialog or in the Glyphs panel in InDesign.

You can speed up the process of hunting for and inserting special characters by using a text substitution app like Texter for Windows (free, open source) or Typinator for Mac (not free). These programs can replace abbreviations that you type with preset phrases. For example, if you wanted to quickly type today’s date you could set up a shortcut that would replace %date with the full date.

I set up a list of text replacements in my copy of Typinator that automatically change certain combinations of characters into IJMES standard transliterated rules. Here’s my list of text transformation rules (all with the prefix -ij, short for IJMES):

-ij’ = ʿ
-ij` = ʾ
-ija = ā
-ijd = ḍ
-ijh = ḥ
-iji = ī
-ijs = ṣ
-ijt = ṭ
-iju = ū
-ijz = ẓ

Here’s a (very) quick example of this in action:

You could set up similar rules for transliteration with different systems (even Franco), or even different languages. Typing IJMES transliterated words for academic papers just got infinitely easier.

You will probably only want to use IJMES transliteration in print because of font encoding issues on different platforms and browsers. For online text you'll have to stick with Franco or something like it.

Libya, obsolete paradigms, and Rip Van Winkle

2009-03-18T00:00:00-06:00

I’m still experimenting with what I’m going to blog about here. I’m thinking of sticking with an odd mixture of technology and academia, which might work out well since I’m considering doing my upcoming thesis on technology in the Middle East, specifically blogging. So, here’s a purely academic post to start things off.

On Tuesday Dr. Lisa Anderson, AUC’s Provost and former professor at Columbia University, came to speak at a faculty seminar at the Middle East Studies Center. She’s a political scientist by training and started her academic career by studying Libya. Her talk focused on the role of traditional social studies disciplines in the study of the Middle East.

General social studies separated into more specialized disciplines in the early 1900s as progressive governments in the US and Europe became increasingly interested in understanding the dynamics of the social “here and now.” New disciplines of social studies emerged with the explicit purpose of improving governance, public administration, and economy. Economists researched and theorized how to stabilize and grow a capitalist economy; political scientists looked at the processes that created a stable democracy; sociologists tackled the dynamics of the changing progressive societies. Social scientists not explicitly concerned with the issues of “here and now” also emerged; those not interested in “here” entered anthropology while those not interested in “now” created the discipline of history.

Over the past century, scholarship in these disciplines has been rather entrenched along this foundational paradigm: research to improve capitalism and democracy. Because of this disciplinary emphasis and paradigm of US/Western-style economics and political systems, any research on other systems of government or economies are always done in terms of the “Western” social science paradigm.

For example, every class I’ve taken on Middle East politics always focuses on the processes of democratization and the persistence of authoritarianism. Elections (fair or not), development of civil society, the emergence of political parties—these are signs of “political development” for a political scientist. Economists look at how liberalized a state’s economy is—how open for investment and primed for capitalism it is. In a way, strict disciplinary social studies of societies outside the US and Europe seem to place other countries on a linear progressive timeline; studies of Egyptian politics reveal how democratized Egypt is or isn’t.

Not all countries—not even Egypt—fit into this standard disciplinary framework. Did Nasser really care if his Arab Socialist revolution fit into a future program of democratization? No way. He didn’t even think of it. Arab political scientists don’t generally look at it that way.

This theme of trying to fit the Middle Eastern peg in the academic square is well pronounced in Libya, which, because international sanctions for the past few decades, has been pretty isolated. According to Dr. Anderson, the case of Libya shows the limits of traditional Western social sciences.

Muammar Qaddafi, the crazy Libyan dictator surrounded by female bodyguards and who made up his country’s name, has ruled Libya for 40 years. He’s been operating with a unique paradigm of political and economic theory that doesn’t jive with the standard Western view. He published a three volume exposition of his political and social views in 1975 called The Green Book. Qaddafi’s view of Libya’s political paradigm is completely different from Western disciplinary political science. For Qaddafi, the ideal political system is more of a radical, romantic, Rousseauian world where each individual represents themselves and has an innate skepticism of the state. His theory of the Libyan economy makes no mention of the market, the main protagonist of “standard” economics. Sociologically, the individual in Qaddafi’s Libya is not the fundamental unit of analysis—the family or tribe is. In practice there are no political parties or parliaments in Libya; each community runs itself with town hall-esque meetings. His social theory seems to be working for the most part.

In the 80s, because of different international incidents involving Libya, including the bombing of Pan Am Flight 103 over Lockerbie, America and most of the international community imposed strict sanctions on Libya, aimed at isolating and punishing them. All relations between America and Libya were effectively destroyed and Libya entered what she called a “Rip Van Winkle” era; life continued in total isolation in Libya under Qaddafi’s political and social theory.

In 2004 (about 20 years later…just like Rip Van Winkle, oddly enough) the sanctions eased and American and the rest of the world began diplomatic and academic contacts again. Because of the sanctions, Libya had missed out on the techno-political revolution of the internet. There were no banks; nobody knew what a credit card was. Dr. Anderson visited several universities in 2004 after this long isolation and found a globe in the center of the reference area of the library—North Asia was still labeled as the USSR.

Don’t go under international sanctions on the eve of a technological revolution. It’s not a good idea. At all.

As the conflict and tension between Libya and the rest of the world began to thaw, American social scientists got excited. They could finally observe Libya’s progress in democratization and economic liberalization; Libya was on the path. Once the sanctions ended, Libya started assimilating into the new world order. It officially apologized for its limited involvement in the Lockerbie crash. It disbanded all attempts at a nuclear weapons program. It was given a rotating seat on the UN security council. Condoleezza Rice even visited and on the eve of George Bush’s presidency, in January 2009, the US and Libya exchanged ambassadors.

However, despite all this apparent liberalization and openness in Libyan politics and the subsequent disciplinary excitement in political science, Libyan experts like Lisa Anderson see a different reality. Qaddafi is following his own trajectory, totally outside the traditional paradigm. He’s not progressing towards democracy—he’s already got a pseduodemocratic Era of the Masses political and economic system and is happy with it. He’s realized that in order to continue with his social revolution, he has to be somewhat involved with the world, and so he presents a facade of integration.

For example, when the Libyans ended their WMD program in 2006 and turned over all their nuclear material, all the machinery was still boxed up in crates. American politicians and political scientists applauded the IAEA for catching and stopping Libya before they could unpack anything. A widespread rumor/theory in both Libya and in academic and political circles, though, claims that Libya never had a program for WMD development. Their infrastructure and level of development couldn’t have handled such a large project. According to this theory, Libya bought the materials from North Korea so they could have something to turn over to the UN and the USA. So, they announced their weapons program, were condemned by the international community, bought some nuclear machinery from Korea, turned themselves in, reduced international sanctions, and improved their reputation in the world.

The same theory applies to the Lockerbie apology. Two Libyans were partially responsible for the bombing, so the majority of the blame was placed on Libya, despite several other claims of responsibility from other international non-state actors. Most Libyans today fully believe that Libya had no real connection to the bombing, yet Libya settled on a large payout with the families of the victims and offered a full apology. Following the apology sanctions were further lightened.

Libya was merely paying cynical lip service to the absurd US-led world order. As Dr. Anderson stated:

If you’re playing the game that cynically, are you really playing the game? Libya is just playing the game; they think it’s just nonsense.

If you’re studying a country that considers the “standard” world political, economic, and social system as an absurd game, can you really fully understand it if you study it using disciplines created by that system for the explicit purpose of improving the system? Standard Western political science is still stuck looking at the “here and now” of the early 1900s. Libya doesn’t fit the social science mold at all—traditional social sciences therefore fall flat on their faces.

I’ve found that the same principle can be applied to other countries in the Middle East. When Egypt held elections in 2005, President Mubarak allowed a second candidate to run for the first time ever. Disciplinary political scientists praised his move as an important step towards liberal democratization. Mubarak won by a landslide through rigged elections and imprisoned his opposing candidate, Ayman Nour.

Great big step towards democracy, or false expectations rooted in a skewed and incorrect social science paradigm?

Do not succumb to economic stupidity

2009-03-15T00:00:00-06:00

Yikes. It’s been a week since I launched my new site and blog and I haven’t posted yet. I think this is partially because I don’t know what I’m going to be focusing on with this blog. Middle East Studies? Technology? Hmmm…

I’ve had this blog post on my mind for a while, though, so I’d better post it. The best part is that it has nothing to do with MES or tech topics.

I’ve felt somewhat immune and aloof from all the American economic troubles. We moved to Egypt in August, a few months after the Bear Stern collapse, but before the September Lehman Bros./Commercial Paper collapse and subsequent bailouts.

Despite the rigors of grad school, I’ve had plenty of time to study up on the causes of this crisis. Having a bus daily 45 minute–1 hour bus commute has given me plenty of time to follow some awesome podcasts. I’ll post about my podcast routine later, but suffice it to say, I feel way more informed about this crisis because of the time I’ve spent listening and studying.

That doesn’t mean I know everything—or anything—about this crisis. I don’t fully comprehend all the crazy financial machinations that got us here. I can follow the news a bit better, though.

Here are some of the best resources I’ve found so far:

NPR’s Planet Money: Amazing podcast that comes out 3–4 times a week. Excellent reporting. Easy to understand. Awesome. Their blog rocks, too.

This American Life: Normally their hour long radio shows highlight a variety of interesting stories on random topics and have little to do with news, but they cosponsor Planet Money and have dedicated two entire shows to the economic crisis, The Giant Pool of Money and Bad Bank. US Treasury Secretary Tim Geithner and Senator Max Baucus even praised Bad Bank in front of a joint session of Congress. It was that good.

The Crisis of Credit: Brilliant 10 minute animation inspired by Planet Money.

Inside the Meltdown: PBS did a great hour long documentary on the crisis. Really good, as well.

So check out those links and get informed. Found any other useful resources out there? Let us know in the comments…

New site launched

2009-03-08T00:00:00-07:00

Two years ago I decided that it would be a good idea to start a personal website. I was always disappointed in the Google results for my name, so I figured having my own web space and domain would help boost my ratings. Yes. I initially started this site for the sole purpose of egogoogling. So much has changed since then…

My first website was ugly. I had just started my foray into web design and barely knew anything—my CSS was rudimentary at best and I used scattered PHP includes to make a pseudo dynamic site. It was a mess.

Ugh. It was pretty embarrassing. I should have just killed it, but I couldn’t because of one tutorial I posted. That one post drives more than 80% of the traffic to this site, and I get almost daily e-mails with questions about it. My old site couldn’t handle the community that built up around it.

Now that I know a thing or two more about design, I’m finally revealing the newest incarnation of AndrewHeiss.com—a fully fledged, Wordpress-based site, with a blog, portfolio, and my general online identity hub.

The design isn’t finished yet; there are a few last tweaks that need to get done. Let me know about any issues, problems, or suggestions in the comments.

Oh, I’m still the top result for Andrew Heiss. Not that it’s hard…

Populating a LiveCycle PDF with PHP and MySQL

2007-10-06T00:00:00-06:00

This tutorial is officially defunct. It is only here for archival purposes. The main script has been consolidated into one PHP class—pdftk-php. Please see the updated tutorial.

I work in the Harold B. Lee Library Multimedia Lab where we check out digital video and still cameras, tripods, external hard drives, digital voice recorders, and let people use $8,000 Quad Core Intel Macs. Expensive stuff…

To insure a “you break it, you pay for it” system, we require all patrons to fill out a loan agreement that we then keep on file. We’ve been using this system for several years and now have more than a thousand forms—all completely unorganized and out of date. We have no way of knowing if a patron has graduated. We have no way of seeing if a patron has filled out a form previously.

So, I volunteered to fix the problem and move the entire loan agreement system to an online database. I had dabbled in LiveCycle and PHP but had never touched MySQL. So I decided to figure it all out.

I’m assuming you already have a server set up with PHP and MySQL. If not, you can download WAMP or MAMP and set up a local server on your computer for testing.

I’m also assuming you have some knowledge of HTML and PHP. If not, search for some PHP tutorials on Google and get a foundation there.

Adobe LiveCycle

Adobe’s LiveCycle PDF Form software works great creating fillable PDF forms and then gathering the form data electronically. Collecting the data is relatively easy since LiveCycle uses semi-open file formats for data storage. For example, if you have an e-mail submit button in your LiveCycle form, a specially formatted XML file will be e-mailed to whatever address you set in the button properties. LiveCycle can then input that XML and repopulate an empty form.

Adobe even sells the LiveCycle Enterprise Suite, which is basically a specialized server made for generating and repopulating PDFs from submitted data. The Enterprise Suite is extremely expensive though

Create form in LiveCycle

Unfortunately LiveCycle Designer does not work like the rest of the Adobe CS3 products. I spend most of my time in InDesign, and from a typographic point of view, Designer is pathetic and a little difficult to work with.

Creating the form is relatively straightforward, regardless of the limitations. Drag text boxes and image boxes from the Library panel to add static text and images. Drag text input boxes from the Library to make fillable fields. Actually laying out and designing the form is not the scope of this tutorial, so I won’t go any further with that.

Data Bindings

What concerns us most is data submission and population with form fields, so we need to set up our fields to work. If you click on a text field, you should have three tabs in the Object panel (if you don’t have an Object panel, go to Window > Object).

The Field tab lets you set some basic properties for the field, like whether or not the field can have multiple lines, the line length, the display pattern for phone numbers or other number patterns, the caption, and a plethora of other things. The Value tab allows you to do some scripting for validation of your fields. I tried getting this to work, but since I’m not a programmer at all and don’t really know Javascript, I gave up.

The Binding tab is the most important for our purposes. By default your field will have a basic name like TextField1. You should change the names of all your fields to a more canonical naming system, like FirstName, LastName, EMail, etc. You can also change the data patterns and formats for text, XHTML, or dates.

Submit Through HTTP

For my purposes, I wanted patrons to be able to fill out this PDF from anywhere and then submit it online, regardless of e-mail accounts. To best do this, drag an HTTP Submit Button from the Standard section of your Library. For this button to do anything, you need to set a URL that will receive the data in the form of HTTP POST.

Begin Setting Up Your Web Application

I made several different PHP files in the process of getting this all to work. First I wanted to verify what the HTTP POST values were before I started trying to process them. I adapted this code from Steve Tibbett, an actual Adobe guy.

Download text for dump.php

Create a file called dump.php and paste this code into it (or remove the .txt extension). Set your HTTP Submit Button URL to dump.php (in my case it was http://localhost/PDFStuff/dump.php) and preview your PDF in Designer. Submit your data and you’ll see all the variables and the raw post data. This step is only to verify that the HTTP POST variables actually match up with your Designer field names.

Now that we know that the HTTP POST variables are actually working, we need to save them to a database. You’ll need to first set up a MySQL table or database. Since I had never done this, I followed some tutorials here, which were extremely helpful in understanding how to actually use MySQL.

Use PHPmyadmin and create a new PDFStuff database with a username and password and then create a table in it with the following command (or use the GUI form in PHPmyadmin to create the table—either way works):

CREATE TABLE `PDF_Loans` (
`id` int(4) NOT NULL auto_increment primary key,
`FirstName` varchar(65) NOT NULL default '',
`LastName` varchar(65) NOT NULL default '',
`EMail` varchar(65) NOT NULL default '',
) TYPE=MyISAM AUTO_INCREMENT=0 ;

You now have a table in your database where we can store our PDF form variables. Paste this code into a file called insert.php and change the PHP variables as necessary, both for your MySQL connection information and your HTTP POST variables (here I just use FirstName, LastName, and Email).

Download text for insert.php

Change the URL of the HTTP POST button to insert.php (again, in my case it’s http://localhost/PDFStuff/insert.php) and try submitting some data with your PDF form. It should work.

Viewing Data in the Database

To actually see your submitted data, create a new php file called view.php and paste this code in, changing it as necessary.

Download text for view.php

This page will take all the data from your database and display it in a table. You should see one record—the one you just added. There is also a column for a link to view the PDF, although the link is blank for now. Add several more through LiveCycle to make sure it’s working.

PDF Madness

If you’ve already had PHP/MySQL experience, all of that was easy. Now comes the tricky part—repopulating the PDF form from the MySQL.

To get this to work, we need to convert the MySQL data into an FDF file, or the Adobe file format for storing form data. We then need to infuse the FDF file into our empty PDF form and allow the user to download it.

Fortunately, someone else figured out the bulk of this, at www.pdfhacks.com. Download pdftk and forge_fdf.php place them in your main site directory. Forge_fdf.php will take your data and transform it into an FDF file while fdftk will insert that FDF into your PDF. All you need to do is add some variables.

Before creating your variables, you need to discover the real names for all of your fields. If you made your PDF in Acrobat, the field names should be identical, but if you used Designer, the official code-based names will be much longer. fdftk can discover those names for you and dump them in a text file.

Place your empty PDF form in your main site folder. Open up a command prompt or terminal and run this command in the site folder, changing file names as necessary:

$ pdftk form.pdf dump_data_fields > form.pdf.fields

Open up the newly created form.pdf.fields file in Notepad and you’ll see the automatic fdftk output, which will look something like this:

FieldType: Text
FieldName: form1[0].#subform[0].#area[0].FirstName[0]
FieldNameAlt: First Name:
FieldFlags: 2
FieldJustification: Left

The FieldName in this case is long and hairy, but we’ll need that full name for the data insertion to work.

Dynamic Data Insertion

Now we’re ready to put all the pieces together. Make a file called viewpdf.php and paste this code in, changing as necessary:

Download text for viewpdf.php

If you don’t want the form flattened (i.e. you want to maintain the form fields), take out the – flatten command.

Notice how the long names had to go in to the $fdf_data_strings array.

To populate the PDF from your view.php table, you need to pass those long field names into viewpdf.php. Technically you would need to change the links in view.php to include the row id number for each row, so the PDF is generated using only the information from that row. Fortunately, we can have PHP and MySQL write all those links dynamically.

We originally set up the database so that every time you insert a record, an auto-id number would be assigned. We can reference that id number to view the PDF for that specific row entry.

You pass the id variable into the viewpdf.php file by adding ?id=1 onto the URL (for example, viewing the PDF for record number three would be viewpdf.php?id=3).

PHP and MySQL can automatically generate that messy link for every record in the table. Just replace <a href="#">View PDF</a> in view.php with <a href="viewpdf.php?id=<?php echo $rows['id']; ?>">View PDF</a>

Open up view.php and see your dynamically generated table. You should have dynamic links that point to viewpdf.php?id=whatever. If you click on one of the links, the code in viewpdf.php will be processed for that row and a PDF will be generated and flattened and downloaded.

Voila!

I was only able to get this to work with text fields, although it is possible to do this with check boxes and other form elements. You’ll have to consult the fdftk documentation to see what variables need to be set for other form elements.

This is a bare bones implementation of PDF population. In real life this is implemented a lot better—i.e. I have my view.php file accessible only after logging in and all the pages are styled with CSS to look nicer.

Hopefully this all made sense. If you want to view the original tutorials I used for this, visit:

MacTech Tutorial—Explains how to do this using an HTML submission form rather than a PDF form. This was the basis for my tutorial.
MacTech Example—Working example of a PDF being populated by HTML.
PHPeasy—Basic PHP/MySQL tutorials.

Using Arabic in InDesign without InDesign ME

2007-09-17T00:00:00-06:00

About a year ago I discovered to my dismay that using Arabic in InDesign was entirely impossible. I wanted to make a type of dictionary for my Arabic 101 students, using an Excel spreadsheet full of Arabic words. When I placed any Arabic text, though, this happened:

While Microsoft and Apple have great right-to-left (RTL) language support built in, Adobe doesn’t. InDesign and Illustrator cannot handle RTL text. Adobe has, however, outsourced their code to WinSoft, who develops the Creative Suite ME (Middle Eastern edition), which does have excellent RTL support, especially through the use of their Tasmeem typesetting framework, recently highlighted in Saudi Aramco Magazine. However, I don’t want to buy the ME version for minimal Arabic use.

Typing backwards

The only way around this is to type the text in backwards: if you want the word alkitaab, you would have to type baatikla and InDesign should show it correctly.

There’s one big caveat though—Arabic letters have different forms depending on where they show up in the word (initial, medial, final, or isolated).

This typing-backwards, faux-RTL works great for Hebrew since almost every letter has the same shape no matter where they are in the word. In fact, InDesignSecrets.com mentioned a script that will take pasted Hebrew characters and reverse them automatically.

Unfortunately, Arabic is more complex. There is a clunky solution—hunt and peck with the glyphs panel. This workaround is not useful for large amounts of Arabic text. If you want to design something with a substantial amount of Arabic, buy InDesign ME. (if you’re desperate, I guess you could do an entire book like this. It would just take several months to get the text done :) ).

Typing with the glyphs panel

The glyphs panel is a great and often underused panel in InDesign. It’s generally used for finding and inserting dingbat characters or other non-standard glyphs in a font. You can even save your most commonly used glyphs for easy access:

You can even type with the glyphs panel, which is how we get Arabic working in InDesign. This method also works for Illustrator and any other Adobe program with a glyphs panel.

To activate the panel, go to Window > Type & Tables > Glyphs. Choose an Arabic font from the list in the bottom left corner of the panel to load that font into the panel. I like working with the Arabic Typesetting font that comes with Office 2007 because of the dozens of alternate glyphs and ligatures that are available. Microsoft has an excellent collection of Arabic fonts as well, found here.

You should see normal Roman characters in the panel. Scroll down until you get the Arabic glyphs. Double click on a letter to insert it at your cursor.

Here’s where the magic starts. Many of the glyphs will have a black triangle in the bottom right corner of the grid box. This means that there are alternate glyphs for that character—in this case, different positions for the letter. Click and hold one of the boxes with alternate glyphs and you’ll see all the different possibilities for that letter.

To type a full Arabic word, insert the appropriately positioned letters in backwards order using the glyphs panel. Here’s a live example of this in action (sorry for the horrible quality):

You can insert alternate glyphs and ligatures too:

If you use a decorative Arabic font, like Microsoft’s Diwani family (found in the arafonts.exe font package), you can use the decorative swashes as well. You can even change the font after inserting the letters to another Arabic font and maintain the letters.

In the end, you’ll have real Arabic text that can be manipulated just like normal InDesign text. It’s a clunky method, but it works, as seen here.

This could all probably be automated with a script of sorts, but I’m no programmer.

If anyone has comments or suggestions (or knows how to make a script for this), leave a comment below…