Extensible Markup Language (XML) tags do not need to be human-readable. In fact, not titling your XML tags in a manner that would make sense to strangers can save a significant amount of storage space and even speed transmission time, advised John King, head of the training firm King Training.
King spoke at the Independent Oracle Users Group's Collaborate 2009 conference in Orlando, Fla. this week. In his presentation, he offered a number of tips of improving the performance of systems that must transfer XML data—most of them around clipping the size of the XML documents themselves.
Cataloguing data with XML has gone a long way toward easing the movement of data among different systems, thanks to the universal use of character streams. The documents that XML can produce are quite verbose though. A peculiar trait of XML is that it was designed to be human-readable, when, in fact, humans rarely read XML documents.
In XML, every bit of data gets its own tag, and every tag is buried in a hierarchy of additional tags to add context around that data. While all the chattiness may not seem harmful when the amount of data you're sharing and storing is comparatively light, "If you transmit 30,000 messages an hour, it adds up," King said.
One thing that can be done is to shorten the tags, technically called elements. An element has two parts—the start tag and the end tag. Each is basically a string of identical characters enclosed within brackets. A start tag is placed at the beginning of the data being encapsulated, and a stop tag is placed at the end of that data. The only difference between the two is the stop tag has "/" character before the string of characters within the bracket.
Elements can be as long as the developer wants, which means they can be as descriptive as possible, such as "<TodaysSalesItems>." Anyone could read that and knows what the data in between the tags has something to do with today's sale items.
But such description is entirely unnecessary because XML markup is rarely read by humans while it is being processed anyway, King reminded the audience. So a tag such as "<TodaysSalesItems>" could be shortened to "<tsi>" thereby saving 32 characters for every time this data element is used (16 characters for the opening tag and 16 for the closing tag).
Many organizations have long had abbreviation dictionaries, perhaps leftover from the days of punch cards when memory was a scarce resource. Such dictionaries offer previously-agreed upon short names for commonly used titles. They could be used as a basis for short element titles, King advised.
XML documents can be further shortened by using attributes whenever possible, instead of elements. Attributes are added on to elements to further clarify the meaning of a particular piece of data, which can save the need to develop a separate element for just some aspect of data. The only drawbacks to attributes are that they can be only be used for a single element and that they can not be further parsed, King said.
Cutting down on white spaces in a document also saves space. Leave out the new line, tab or blank space commands cuts bytes from the document. The data may not be as easily readable for the humans, as all the data is mushed together in a single line, but it is just as easily read by the machine.
Another place to look to shave bytes is to strip out all the unneeded sections of the Character Data (CDATA) section of the document, King advised. CDATA is the part of the XML document that is not parsed. It usually is used to stash notes about the document itself. While this is helpful for someone trying to understand the information, it makes no sense to include all CDATA information in each transmission of the information.
In order to show the storage and transmission savings that could be enjoyed by these techniques, King took a sample of XML'ed data from Oracle documentation, and substituted in shorter element names. He reduced a set of tags that was 313 characters in length to a set 215 characters in length, which is about two-thirds the original. He also made a set where he substituted attributes for some elements, reducing the size of the set even further, to 179 characters. He then encoded 30,000 records with each set of tags, so that they could be compared in terms of size and how quickly they could be committed to a database.
The regular data set, in a flat-file format, was 39.4 megabytes (MB), while the one with the shorter tags was 28.9 MB. The set with the attributes was 27.3 MB. Using a standard laptop with 4 gigabytes of working memory, it took 29 seconds to store original set of data in a database. It took 23 seconds to store the shortened set, and 21 seconds to store the set with attributes.
The only quirky finding King came across was that once in storage, the data set with attributes took up more space (20 MB) inside the database, than either the dataset with the shortened tags (11 MB) or the one with the original tags (14 MB).
King also noted that, when it comes to storing data in a database, the most efficient method, space-wise, is to shred the data, or pull each data out of its XML-tag encasement and store in a separate database field. Using this approach, King found that all three datasets took exactly the same amount of space in the database—8 MB. It took 11 minutes to shred each data set, however. So shredding saves space but consumes time, King noted.
While the speed of XML processing can be picked up by slimming down the XML data itself, King also offered a number of other ways to speed XML data transfers. Of course, vendors offer a wide range of XML accelerators, or appliances that strip the XML tags as they are sent from the data source and add them back in at the destination. This approach reduces the flexibility that comes with using XML, however. Another approach is using the JavaScript Object Notation (JSON), a lightweight data-interchange format.
One thing to take a look at is if the data is being validated against an XML schema, and, if so, if it is necessary. By validating an XML document, you are checking to see if it is in the appropriate structure, as defined by its schema. While this is a valuable function when receiving data from other parties, checking data when it is being moved about internally, however, consumes uneccessary processing power.
Also, King advised to look at what is being transmitted and question if all the elements need to be transmitted. For instance, a description of some item that is being ordered through a parts requisition system does not need to be included—only the part number might be necessary. In many cases, the complete set of information is sent in order to validate the data-mark-up. XML parsers can not validate XML files unless the entire structure of that schema is in place.
"Being well-formed eats our lunch, in terms of performance," King said.
Posted on May 05, 2009 at 7:05 PM3 comments
We've been covering Twitter quite a bit recently, though, like the Internet itself, it is something best experienced personally. The good news is that, in its most basic incarnation, Twitter is about the easiest Web application you could ever use — even if it is, at first glance, a bit obtuse to outsiders.
Here is how to get started.
First, go to Twitter's home page, you'll find a "Get Started — Join!" box, and pick a user name and password. After you fill out and submit everything Twitter wants you to, you are presented with your landing page. Bookmark that page; that's the one you'll want to return to whenever you want to check out Twitter.
In the middle of the page is a small box, which you can click and then fill in with whatever it is you want to say. Try to say something interesting. Keep in mind you have only 140 characters to work with (the page helpfully offers a counter that will let you know how many characters you have left after you start typing).
When you're finished your missive, hit "update." There you go, you've just tweeted.
For anyone to see this message, of course, you'll need a few "Followers." These are the people who can read your messages from their own Twitter accounts. The easiest way to get followers is to let people know you are on Twitter. Send out your Twitter name to colleagues and friends who are already using the service, so they can add you to the list of people they follow.
You'll also want to keep tabs on what other Twitterers are doing. When you "Follow" someone, their messages will appear on your Twitter landing-page. On the top-right hand of the page, you'll find a "Find People" search, which you can use to find other Twitter users by name and then add them to your list of people to follow. Also, scope out the the GovTwit site, it keeps a list of government personnel who are already tweeting.
And this is where the fun starts: As you add more followers, and get more updates from your group, your landing page will start to resemble an ever-flowing stream of updates, giving you a near-real-time completely-personalized ticker feed of what your peers are up to. This can be useful for your job, or just plain entertaining, depending on the kind of folks you follow (You'll have to keep hitting the browser's refresh button to get that ticker-tape effect).
If you want to reply to someone's else's message, of just send a message to their attention, put the recipient's Twitter handle, prefixed with a "@" sign, into the message, i.e. "Hello! @Govcomputernews" would send a message to Govcomputernews. The recipient would see the message in his or her replies page (which is a link off the main landing page with the account holder's handle). Note: These reply messages can be seen by anyone who is following your account. You can also send a direct message that can only be seen by the recipient, but that recipient needs to be following you for you to mail that person a private message.
Another way to get the word out about what you are Twittering about is to insert hash tags into messages. A hash tag is the "#" symbol affixed to a word that describes what you are writing about. That way, your message will be picked up by services that periodically do searches on widely-used hash tags and aggregate the results, as well as by Twitter's own search engine.
For instance, Steve Ressler declared "#GovLoop" would be a hash tag that his government's social networking site, GovLoop, would use to identify those messages that would be interest to that site. The home page of that site now has a link to the results of a real-time search of all the messages tagged with #GovLoop.
And that's about it. There are other aspects about running a Twitter account that could be worth attending to over time, such as setting up a private account or customizing your profile page, so check the Twitter Getting Started page for more tips.
Note: For a rundown of more advanced desktop clients that can be used Twitter, check out the GCN guide, here.
Posted on Apr 14, 2009 at 7:05 PM0 comments
It's official: Twitter has made it to the top ranks of government. Yesterday, the head of the U.S. Armed Forces, Navy Adm. Mike Mullen, chairman of the Joint Chiefs of Staff, started tweeting.
According to a survey by research firm CommStat, almost 10 million people were using the microblogging service as of February. While 10 million is still small potatoes compared to, say, Facebook's small nation of 200 million users, Twitter's increase in users is quite impressive, up 700 percent from a year ago.
Moreover, "Tweeting" is not just something that the kids are doing: The majority of Twitter users are older than 35.
Who tweets in government? A surprising number of folks, and they are using it for all sorts of reasons. As part of its Twitter Grader service, marketing firm HubSpot compiles the numbers of Twitter users in major cities around the globe. If these stats are accurate, D.C. seems to be catching on to the phenomenon, right behind the digerati in other tech-savvy cities such as London, San Francisco, and Austin, Texas.
"We never thought we'd say this, but some of the best and most innovative new media experiments going on right now on the Internet are coming from the U.S. federal government," Business Insider's Silicon Valley Insider blog gushed not too long ago.
We suspect that Mullen himself will not be tweeting his deepest, darkest middle-of-the-night worries about, say, Iran or North Korea. Thus far, his tweets seem to official announcements from the Joint Chiefs of Staff office. And, given Mullen's daily schedule (we understand he is touring India and Pakistan at the moment), we wouldn't be surprised if a ghostwriter may actually be posting them, a favored tactic among busy celebrities.
But official use works, too. The format is flexible enough that it can be used for a wide variety of communiqués — from the official to the personal. As long as it can be stated in 140 characters or less, it can be added into the ever-flowing stream of tweets that define the day.
The Defense Department is not the first agency to use Twitter. NASA's Jet Propulsion laboratory found the service a great way to provide updates on the Mars Rovers. More than 43,000 Twitter users have signed up for updates from @marsphoenix account. Elsewhere, the National Institutes of Health posts health-related dispatches. The U.S. Geological Survey posts earthquake and tsunami warnings, while the Food and Drug Administration posts updates on food recalls.
At the state and local level, the Los Angeles Fire Department posts updates about its calls on Twitter. Both the New York State office of the Chief Information Officer and the Utah State Library Digital Library Services post about the latest computer services that their states offer, to offer two examples.
Congress has taken to Tweeting in a major way, what with 19 senators and 50 members of the House of Representatives all using the service (two volunteer-run sites, Congressional 140 site and TweetCongress, capture the latest posts from all these elected officials).
Twitter is also used by government personnel as a way to explore and document their respective fields. Bev Godwin, the director of USA.gov and the White House's new-media guru, used the format to report on some of the sessions at the recent Government 2.0 Camp. NASA astronaut Mike Massimino is chronicling his training for the fifth and final space shuttle Atlantis mission to service the Hubble Space Telescope, according to NASA. And Dan Mintz, the former the chief information officer of the Transportation Department, is a dedicated observer of all things tech through his feed.
And this is just the tip of the seemingly ever-growing iceberg. Bearing Point’s Steve Lunceford compiles a list of government Twitterers in government agencies and the government contracting community, called GovTwit. At last count, 1,060 names are on the list.
Steve Ressler, who runs the GovLoop government-oriented social networking site, started a discussion thread asking everyone to name their favorite government bloggers. Increasingly, conversations that used to take place by e-mail or around the water cooler are ending up on Twitter.
"The main source of most of our new traffic is from Twitter," Ressler commented in an interview with GCN affiliate publication Federal Computer Week.
Posted on Apr 09, 2009 at 7:05 PM4 comments