Thursday, November 13, 2014

Odds and Sods

FileMaker 13 is back at Tech Soup. A one-year subscription to FileMaker Server is $649. A full license for FileMaker Pro (the desktop client) is $194. Unfortunately, they don't offer non-profit pricing for FileMaker Pro Advanced at TechSoup, but you can inquire directly at FileMaker, where there are frequent deals. During November they are offering a 2 for 1 deal for FM Pro and FM Advanced when you buy direct.

PITA of the week: The new OSX Yosemite transmits search data by default to Apple, Microsoft, and god-knows-where. This is a reversal from previous versions of OSX. There is a fix.

Powershell is turning out to be pretty amazing. There is an entertaining introductory video series from Microsoft Virtual Academy which includes Jeffery Snover, the original PowerShell author who explains why they do things the way they do.  

Ozzie Zehner is a green-technology skeptic, in the sense that he suggests that our infatuation with alternative energy like photovoltaics and windmills really perpetuates the energy status quo. His top suggestions for committed environmentalists; empower women and girls.
Green Illusions pioneers a critique of alternative energy from an environmental perspective, arguing concerned citizens should instead focus on walkable communities, improved consumption, governance, and most notably, women’s rights.

PowerShell: Moved to new blog

PowerShell was clogging up Tech For Non-Profits. For the past several weeks I wrote a bunch of posts about using Windows PowerShell to access web APIs.  This has escalated to the extent that I thought it would worth an entire blog, so I've moved these posts to the brand-new shiny PowerShell Notebook. where I'm groping around for PowerShell mastery.  So far I have code to show how to access the following APIs: 


There are also entries about scripting FTP commands, and some general discussion of text processing. 

Take a look at

Tuesday, September 16, 2014

A Database for Grant Research

I put together a grants database screen (click to view full size) to consolidate information for funding sources, and to track dates and interactions.

It is definitely an evolving project, but contains the basic information need to contact the funder, the deadline dates involved, the funder's areas of interest, and the typical range of a grant award.

So far, I've been concentrating on foundation funding. Many foundations typically ask for a letter of interest before you put together a full proposal. So, I've included multiple date fields, a deadline for a letter of interest, a deadline for a full proposal, and a date when they announce their award.

Originally I thought that this database would be mostly for research, but after working with the online grants database, Grantstation, I think I will reserve this database for funders that I really expect to submit to. Some ideas for future enhancements include:
  • Links to standard "boilerplate" paragraphs that are used in an application. 
  • Links to edit the proposal or letter directly in Word. 
  • Links to the PDFs of the proposal. 
  • Reports that create a grants calendar. 
Before anyone comments that "you should really use X software" for this purpose, I just want to say that I've used several in the past, including DonorPerfect and Blackbaud, and evaluated many others. Right now, I'm in the process of rethinking my entire workflow automation from the ground up, and this very lightweight approach is just what I'm looking for. Plus its in FileMaker, so I can run it on my Windows machines at work, or my Macs at home.     

Odds and Sods: Grantstation Membership and Training Opportunities


Grantstation is a subscription-based database of grant opportunities. You can purchase a l year Grantstation Membership for $99.00 per year if you are a member of TechSoup, on September 23 and 24.

The free Grantstation 1-hour orientation webinar is well worth the time. I learned, for example that their federal grants database is a front-end using data from    

Tech Soup is also sponsoring a free webinar by the CEO of Grantstation Cynthia Adams on September 18th.

Funding Rural America :
When has it ever been easy to secure funding for nonprofits and libraries in rural communities? Is there a way to level the playing field so organizations in small towns, counties, or boroughs without large metropolitan areas can compete for both government and private sector grants? What are the other options for financially supporting a rural project? Are collaborative efforts worth the effort? And, of course the biggest question: who is funding rural America?

These, and other relevant questions, along with current trends affecting rural funding will be addressed in this free, two-hour webinar presented by Cynthia Adams, CEO of GrantStation.

Finally, it is rare that we find a Grantmsanship Seminar in our neck of the woods (northern New England). But  there will be a five-day Grantmanship training seminar located in Barre VT. December 8-12, 2014

Hardware and Software is  another possible source for refurbished computers, (or a place to donate your old gear). They are one of the refurbishers for the Techsoup Refurbished Computer Initiative.

Techsoup has wonderful deals for software. A one-year subscription to FileMaker server is $209, and a subscription for FileMaker Pro (the desktop version that runs on a Mac or PC desktop) is $65.00. Visio Professional is available for $29.00.


Here is an interesting academic discussion of luck, and how to have more of it. Thanks to Jeff Duntemann for the link.

Tuesday, August 12, 2014

Rebootolator: Execute a Remote Linux Shell Script from Windows

Ok,  so, your mission, should you decide to accept it, is to restart mySQL and Apache on a remote server. This restarts a balky web site hosted by Apache, and also restarts a mySQL server which is used for a back-end for Drupal.

You want to execute this from your Windows computer.

The target computer runs CentOS 5.6 This is an (ancient) Red Hat Linux derivative, running (ancient) mySQL and Apache.

I ended up using PLink called from a Windows .CMD file to execute a bash shell script.  The shell script looks like this:


#!/bin/bash -p
# Rebootolator - Reboots Apache and mySQL on a target Server
# LK Microdesign June 25, 2014
export TERM
echo 'Rebooting Apache and mySQL on myServer'
echo '-----------------------------------'
echo 'Restarting mySQL'
/etc/init.d/httpd restart
echo ''
echo 'Restarting the Apache web server.'
/etc/init.d/mysqld restart
echo 'Reboot procedure completed'

Note this script is not stored on the target server, but simply put in the same folder as the windows cmd file on my windows box.  

Now for the Windows command file: 


:: Batch file to restart services on myServer
:: Restarts mySQL and httpd 
:: Uses the Rebootolator shell script
:: LK/Microdesign August 12, 2014  
@echo off
plink -ssh -m -pw mypass

pause >nul | echo Press any key to exit. 

So, lets deconstruct the Windows Reboot.CMD file.
The first four lines are comment lines. Turns out, you can use two colons to preface a comment in Windows, (who knew?) instead of REM.
Line 5 turns off output to the screen.
Line 6 clears the screen.
Line 7 and 8 put in blank lines.
All the work happens on line 9, using the PLINK command. PLINK is the command line version of PUTTY, a free open source terminal program for Windows workstations. Both PLINK and PUTTY are pretty wonderful and highly recommended if you need to access Linux machines from Windows.
-ssh means "use the secure socket layer protocol to log into this machine" is a administrator's account on the target machine,  probably the root account.
-m is the name of the shell script (above) that needs to run on the target machine.
-pw mypass is the password for the account used to log into the machine.

Deconstructing the script:
#!/bin/bash -p  just means this is a BASH script
The two commands that actually restart the mySQL server, and the Apache server are: 
/etc/init.d/httpd restart
/etc/init.d/mysqld restart

The rest, (the echo commands) write out what  is happening at the command line. The Term command is my attempt to avoid a harmless error message that occurs when the script starts to execute.

Since I didn't realize I could host the script in my Windows folder, I originally though I'd have to log into one Linux box, and then execute the script on the target box.  Turned out the whole thing was simpler using PLINK, which is the equivalent of SSH and SSHPASS programs used to access remote machines from the Linux command line.

Thursday, August 07, 2014

Tech Friday: FileMaker Resources

Random FileMaker-related resources: 

1. You can create an alternative icon for FileMaker 13. When you have multiple FileMaker versions installed, it sometimes is difficult to distinguish between them. FM12 and FM13 have virtually identical icons with the same color schemes. Here are alternative icons that can be installed, for both Win and Mac from HomeBase Software. HomeBase has a ton of of technical information on their web site.

2. In support of a project to integrate SmartyStreets with FileMaker, I've been doing some additional research on JSON, (Javascript Object Notation), which is a simplified version of XML.

3. Coding Standards for FileMaker

4. Modular FileMaker: is shared library of FileMaker functions. There are huge community-developed libraries for other languages such as PHP and Python. These folks are attempting a similar idea for FM. I've downloaded their JSON module, and am experimenting with it. Other examples include a nifty SQL query generator, and and another interface to Mailchimp.

Custom Functions are a way of adding small chunks of user-defined code that can be called within a FileMaker script. Brian Dunning is the guru here and curates the largest library of custom FM functions on the web. He also has sample data sets available for the U.S., Austria, Canada, and the UK. Five hundred records are free, and a million records are available for ten bucks.

5. If you need a 100,000 records or so, you could also download the database of public and private schools available at the National Center for Education Statistics. This includes demographic data as well as mailing and location addresses for schools. The data is fun to play around with. You can give yourself some sample exercises in FileMaker. For example:

6. What percentage of public school students are eligible for free or reduced lunch in your state? It is 38% in mine. Poking around in some other states, it looks like that isn't unusual; in many states it is 40-60% or more. The lunch program is often considered a proxy for the family poverty rate. But maybe that's another discussion.


Sync Google Apps with Microsoft OutLook 2013

I installed Microsoft Outlook 2013 the other day, along with the rest of Office 2013. I'm only about two years late, right?  Office 2013 offers little over Office 2010, but there is a somewhat cleaner design (OSX 10.7 anyone?) and really nice animations when windows open and close. Forgive me for saying this, but the interface is more Mac-like, even though it came out two years before the recent updates to OSX and IOS 7.

I originally had OutLook eMail synced to my Google Apps email. Then I wanted to also use the Outlook calendar and have changes made there reflected in my Google Apps calendar which is the group calendar that we use in the office. A quick search found the Google Outlook synchronizer, an application that sits outside of Outlook and runs interference between Outlook and Google apps.

The nice thing about this is that that it doesn't require you to put in all kinds of IMAP information into your Outlook simply give it your Google Apps eMail address and off it goes and downloads everything from Google Apps, creates a new profile within OutLook, and then uploads everything into mailboxes in Outlook (see screenshot). If it works, it will great. I includes your contact lists, shared calendars, co-workers calendars, notes, and tasks.

The poor mans's Exchange Server? 

If you use Google Apps in the office, and have Windows computers, you could have this running on each machine, and essentially use Google Apps as a back-end Exchange Server.

Monday, July 28, 2014

SmartyStreets - Validate Mailing Addresses

Some months ago I submitted a mailing list file to our mailer (we use Quad), and I was somewhat taken aback to get their report that over 5% of the addresses that we sent to them were invalid or incorrect.. I was thinking this wasn't too bad a figure, but they informed me that "people who knew what they were doing" would achieve valid rating of close to 100%. My thought was to do some "pre-validation" before sending the mail file to Quad. Turns out this can get expensive. Then I found SmartyStreets.

SmartyStreets is a web-based address validator for U.S. addresses. Using the US Postal Service official address database, SmartyStreets will validate any address that you send to it with varying amounts of correction.  Addresses are classified in a number of different ways, including:

  • Nomatch             The address is invalid.
  • Mailable              The address is valid and can receive mail. 
  • Mailable-Vacant:  The address is valid but vacant
  • Match-Inactive.   The address is valid but inactive. 
SmartyStreets has a one-off web interface which allows you to validate addresses one at a time on the fly. It will supply a nine digit zip code for valid street address, and it will suggest addresses nearby if you submit an invalid address. In short, if SmartyStreets returns an address, it will most probably be mailable. That's what I'm hoping, anyway. I just sent a new file to Quad, and we'll see what the accuracy is.

Click to Enlarge.

The web-based interface returns, not just a corrected address but additional information such as lattitude and longitude of the address, whether it is commercial or residential,and the address's time zone and congressional district. 

Smartystreets will process lists of addresses. You can simply paste in an ASCII comma-delimited list or an Excel file containing an address list, and SmartyStreets will return the list with in a mail/no-mail format, or with about 30 fields of additional information. 

SmartyStreets frequently donates their services to churches, schools, libraries, and many non-profit organizations. They have extensive documentation that explains how the validation process works, and what the results of your file can contain. And they have an API (an application programmer's interface) which will return results in XML or JSON format.  

A great addition to the mailing toolbox.  

Friday, June 13, 2014

Alabama Eye Bank runs on FileMaker Pro.

Over at  Tech for Home Healtcare,  I've described how the Alabama Eye Bank uses FileMaker Pro to manage the process of receiving donated corneas thorough finding a recipient and scheduling the surgery. It is an amazing application that shows the cross-platform versatility of FileMaker, hosted on Windows servers, and deployed to Mac workstations, iPads, and iPhones.  

Thursday, June 12, 2014

Refurbished Desktop Computers

Refurbs are for when you have more time than money. I'm not sure about the exact figure, but in many cases, I think I've ended up spending several hours per unit getting a refurbished computer back online after a hard drive failure, or just having to spend hours updating Windows and Office so that I'm confident getting the machine on the network.

We got several "really good deals" from NewEgg, for refurbished Lenovo desktop computers at $214.00. These appeared to be of "office quality", included Windows 7 Pro, and were nicely finished. Unfortunately, we have had 2/3 of the Western Digital Blue hard drives start to fail at some point. This has created no end of extra heartache for the users and an enormous amount of work for the IT staff.

NewEgg has been fine on returns, however, providing UPS shipping labels, and RMA procedures over the web. much for NewEgg.  We're looking at alternatives.  (we have more time than money).

Techsoup has Dell refurbished computers that are prepared by a third party. For example:

Dell OptiPlex 755 Core 2 Duo Windows 7 Desktop 2.0 Ghz - 2.6 Ghz 
Min of 160Gb drive
Min of 2Gs RAM 
Windows 7 Professional 64 bit. 
Also includes: 
Office 2010
Adobe Flash,Reader 

One advantage here is that if you need licenses for Windows 7 and Office, they are included in the price. You would spend the $286.00 on those if you bought at retail, and maybe quite a lot less if you have a Microsoft Open, agreement. But, it like getting the hardware free.

The Dell Outlet looks promising with several machines in the $315-$390 range which still include Windows 7 as opposed to Windows 8, and have at least 500Gb drives, and 4 Gigs of RAM. These have more up-to-date processors than the Techsoup machines, and are certainly not as old. Most Dell Outlet machines were either not delivered, or were taken back within the warrenty period.

I've had solid results with Dell Outlet computers at the workstation and server level; mixed results with standard desktop machines, and a real disaster with older SX-series Optiplexes.  The best seem to be the larger ones; towers or mini-towers. Smaller machines, "mini-desktops" may have suffer from the suboptimal cooling, and the older components may have reached their design end-of-life earlier than those installed in a larger case.

One thing we have often found is that dual monitors are wonderful, and this is something that I would recommend for anyone as a matter of course. If you need an extra monitor card, these can be found from NewEgg starting at around $35.00. Best to wait until you have received the machine, because there can be variations in the slots, and the available adapter space that aren't evident from the web page.

On the Mac side, I've purchased several Macbooks, iPods, iPads, from the Apple Store. These have always worked flawlessly. The Refurb store has a 21.5 inch iMac for $1099, which is the model from September 2013. The cost is only $200 or so less that of a new, similar iMac. It includes 8 gigs of RAM, and 1 terabyte hard drive, and of course the Mavrick OS, and iWork. If you've got more money than time, and just want to get to work, this might be the way to go.

Tuesday, May 06, 2014

FileMaker: A Quick and Dirty SQL Query Screen

How to create a quick and dirty SQL screen for a FileMaker database:
1. Create a data table. I call it SQLWB. Give this table two text fields: "Query" and "Result". Both of these will be text fields.
 2. Modify the layout I call it SQL Workbench.
 3. Resize the two fields to show a larger amount of text. I add a scrollbar to the end of each field

4. Create a scripts to run the query.  I call it RunSQLQuery.

Set Field [SQLWB::Results; ExeuteSQL (SQLWB::Query;"";"")]

5. Create a script to clear the two text boxes.  I call this script ClearResults.

Set Field [SQLWB::Query;""]
Set Field [SQLWB::Results;""]

6. Add two buttons to the layout; one for each of the scripts.

The result will look something like this:

Note that this isn't a full-blown SQL implementation, it will only respond to SELECT statements. (You can't INSERT or DELETE, for example.   Also, if there any problems with your SQL syntax, the result box will only show a question mark.

Thursday, April 10, 2014

Odds and Sods

Miscellaneous Items

I've been upgrading my Dell Optiplex 620, (this of the noisy fan and the overheating case which always needs to have the top left off. ) from Windows XP to Windows 7. I used an image downloaded from the Microsoft licensing web site, and, I think this must be original version of Windows 7 that came out in 2009 or whenever, because it has applied over 150 updates, to the newly installed version, which has taken over four hours to have ready to use. Cripes, haven't they even heard of version control, or Git?

After casting around for a cloud-based web hosting company,  I'm leaning toward Linode. To start with, their documentation is terrific.

And, speaking of Git if you are trying to get your head around it, here is a great beginner tutorial that describes the theory of Git and doesn't just give another rundown of all the commands.

Visual Inspiration at Urban Sketchers.

Trello is an uncomplicated way to manage projects. There is an e-Book about Trello, Trello Dojo published by Daniel Root and published by Leanpub

As a professional nerd, occasionally I’ll run into a website, utility, or service that has potential to help the day-to-day lives of people not only the IT field, but in about any line of work. Or, perhaps even people in no line of work. Trello is just that sort of service. Their website describes it: “In one glance, Trello tells you what’s being worked on, who’s working on what, and where something is in a process.” Being free, easy to use, and constantly updated, giving Trello a try in your next project or process is a no brainer. You have almost no risk and a lot to gain.

The book comes with pointers to more than a dozen Trello templates; projects that are already set up. There is also discussion of feeding a Trello board with inputs from other sources, such as an RSS feed or eMail. An example of such an application is the Poor Man's Customer Service tracker, which takes eMails sent to your customer support web address, and automatically creates a Trello card for each issue.

Sunday, April 06, 2014

Tech Friday: API Programming 101

I've working on an add-on application for the Shopify e-Commerce web service, that allows you to have a web store completely in the cloud. What we're trying to do is to have the cart check-out function call a custom application which calculates shipping costs, because we have a slightly idiosyncratic way of dealing with those that isn't accommodated by the standard Shopify functionality.

Shopify provides an API (Application Programming Interface) to their web site which allows developers to add extra functions to a customer's web shop. Entry to the API is
provided in several different ways:
  • External Apps: Shopify calls an external function hosted at a publicly available web address. Presumably the same app provides functions for multiple customers and shops. There are bunch of these advertised on the Shopify web site. Some are free, some are supported by Shopify and some are built by third-party programmers, who make them available for $5.00 - $60.00 per month.  
  • Internal Apps: These are similar to internal apps, but are available for a single store. 
  • Embedded Apps: Haven't quite figured this out yet, but my guess is that somehow this allows an app to be embedded in a single store, and hosted within the Shopify web server, thus obviating the need for a separate web server, and public web address.
I quickly wired up our shipping calculation in PHP. Its about 12 lines of code. The problem has been to try to find how to connect it to our Shopify check-out cart. Along the way, however, I've found some interesting tools.

It is a little ironic that some of the best PHP programming tools seem to exist primarly in Windows versions, but that most of the deployment of PHP happens on Linux/Apache web servers.  Ideas:

Notepad++   The favorite of my boss, but as I said to him, "its like writing with a stick in dirt".  I use it all the time though for general editing of ASCII files.
JetBrains phpStorm I have the thirty-day trial for this, which also runs on the Mac. Looks promising.
CodeLobster for PHP Just found this free PHP editor which also has some support for javascript and CSS.

To create the publicly available web address that will host my app, I got an account with Heroku 

To examine calls to and from web services, I downloaded the the Firefox Plug-in HttpRequester.

To examine the web traffic, I use Fiddler.  To see web traffic in greater context (and to see all the data you're sending to the NSA...)  I use WireShark.

All of the above are Windows applications. Other suggestions cheerfully accepted.  (For example, on the Mac, WiresShark uses an external Xquartz graphics library, which has to be launched separately. The whole thing seems a little flakey, but it works, all the same.)

Friday, March 21, 2014

Tech Friday: XP is over, Long Live 7

With the impending demise of support for Windows XP due in April, one solution I investigated was replacing an entire workstation with a refurbished machine supplied by NewEgg. They had a several Lenovo units for sale for $219, including shipping. These included a 250 megabyte hard drive, a 2.8 or 3.0 dual-core Intel processor, and 4 Gigs of RAM, and Windows Professional 7.

We received the unit promptly, and as I unboxed it, I noticed that it appears to be an IBM-branded unit built by Lenovo of about vintage 2009. This is a little older than I would have hoped, but the Windows 7 setup seems to be going smoothly so far. The unit has a Windows "Microsoft Refurbished" sticker on it which has a valid Windows 7 code. There was also a "Made for Vista" sticker on the case, which I've scratched off. Yikes.

So far, main irritation is that the case is so small that there is no room for a full length expansion card. We normally have dual monitors for everyone, so expanding the single VGA motherboard connector may prove to be a challenge. Still, a machine with the hardware specs above, and Windows 7 for a little over $200.00... that is inexpensive. Comparable new machines were in the $600 range.

I've had mixed results with refurbs at least with Dell. A couple of Dell SX series went belly up, and one was dead on arrival. A recent Optiplex 620 series seems to overheat. I've had two full tower refurbs from Dell which have been just fine, as well as an Inspiron laptop. So, let's hope the Lenovo, er, IBM is solid.

But back to XP. Isn't it odd that although Microsoft is trying to entice users to move to the cloud, and subscription-based software, it can't seem to be in a position to offer a subscription-based support plan for $20.00 or $50.00 a year for XP. John Dvorak outlines a plan here.

I always thought Microsoft was in the business of making money. If you are a shareholder, go to the next meeting and ask the bigwigs why Microsoft is leaving what can amount to billions of dollars on the table regarding Microsoft XP. 
Microsoft is too dumb to realize that it can effortlessly monetize Windows XP using the beloved software subscription/rental scheme the company keeps discussing, but apparently has nothing but trouble trying to implement. 
I am going to describe a multi-billion dollar idea that Microsoft must consider. 
There are still approximately 500 million XP users—an estimated 29 percent of the computers in the world. Many do not want to upgrade to anything new. They are happy campers. 
These folks, according to a variety of reports, include many banks and most of the ATM networks. Smart corporate money knows that if you have something that works great for a single application you do not swap it out. You run it until something comes along that would save you enormous amounts of money. This is not happening with banking software.

Friday, March 14, 2014

ClearOS: A Linux-based Windows SBS Replacement

In my ongoing project of trying to clone a Redhat Linux server, I ran across a help file that was written for an operating system called ClearOS. I assumed that this was another Linux distribution, and ignored it at first but then, while waiting for another installation, I spent some time reading the web pages.

ClearOS is a combination of a core Linux distribution based on Red Hat and CentOS. It includes a complete set of applications to provision an entire office. Perhaps the main advantage is that it takes what are usually a number of several different disparate Linux-based programs, and it puts a slick web-based management front-end them. ClearOS is very modular; you can make things as sophisticated or simple as you want.

To get a closer look, I downloaded and installed the ClearOS Virtualbox demo. (The only glitch was a problem with the 64-bit demo; I re-downloaded the 32-bit version and that installed perfectly on Virtualbox on my iMac.)

Setup is accomplished by a wizard that walks you through a sequence of steps to install the software, connect to the internet, configure the firewall and configure additional services.

If you want to see how ClearOS looks without worrying about the installation, you can "manage" a virtual server with a Live-Demo.

The screenshot shows options for backing up local workstations.

ClearOS offers a number of different versions and support levels. You can download and run the community edition for free, a choice that I might consider to replace a Windows SBS 2011 server if there are no processes on the server that are dependent on Windows. You can install it and run it on your own dedicated hardware. (They don't recommend running the whole thing in a single virtual machine).

Or you can run it on a ClearOS hybrid appliance. These require the ClearOS Professional version which is a subscription-based support plan. The supported version can also be run on your own hardware. It includes certified and tested versions of all of the applications so that they are guaranteed to work together.

Years ago there was the Cobalt Qube, a single box which provided eMail, file and print services in a single cute box. (You can still find them on eBay). It was a great way to get an "instant network", and I was sorry to see it discontinued. The ClearOS options provide a similar instant network, and would be suitable anywhere a Windows Small Business Server might be considered.

Friday, February 28, 2014

Mining the Social Web

O'Reilly recently held a 1/2 price sale for electronic versions of some of their newer books. I bit on Mining The Social Web 2nd Edition by Matthew Russell. Having just started the book, I'm not yet in a position to really comment, but I was intrigued by several aspects. Basically the book is about using the Python programming language to access social media sites through their API (application programming interface).

Instead of having the reader set up Python on your own machine, he created a virtual environment that could be run on any platform using a program Vagrant. The VM uses a Python teaching framework called IPython Notebook which allows you to have interactive code examples embedded in text.

The combination of the virtual machine, Vagrant and IPython Notebook obviates the need to document and maintain all the possible permutations of each platform (Mac/Windows/Linux) and version of Python, all the related ancillary libraries, and whatever is needed to install and maintain updates. 

The repository for all of the Python code used in the book and the setup for the Vagrant virtual machine is on GitHub. GitHub is a cloud-based version control system, with a lightweight social media overlay that allows people to collaborate on programming projects. 

Vagrant can use different virtual machine software, but it just happens that the setup for the iPython Notebook provided uses the command-line version of VirtualBox. Vagrant installs an instance of Ubuntu 12. with all of the provisions for hosting Python and the iPython Notebook, as well as a web server which makes this available on port 8888.  So, the installation sequence for Windows is:
1. Download and install VirtualBox if you haven't already.
2. Download and install Vagrant.
3. Establish an account on GitHub
4. Download the GitHub client. This includes two applications, the GitHub terminal, and the GitHub GUI Manager.
5. Using Git, "Clone" the GitHub directory for Mining the Social Web Second Edition. This makes a copy of the GitHub directory on your local machine.
6. Start a command line session, and CD to the MTSWSE directory (whatever you've named it) Run vagrant up. This starts the creation of the virtual machine, and the full provisioning. It is about a twenty minute process. And, to be sure, given all of the output that it generates, it looks as if a multi-hour job has been automated.

A couple troubleshooting tips. If Vagrant looks like it is installing "the default VM",  or if you see a reference to PROCESS32 you've somehow missed the GitHub part... maybe the directory is wrong? What you should expect to see instead is PROCESS64.

If correct, the provisioning process will end with a line
DEBUG: Exiting. 

The real sign of success is to open a web browser and navigate to http://localhost:8888
That will bring up the main page of the IPython book..

My favorite error message to date:
DL is deprecated, use FIDDLE.  

The Vagrant documentation refers to another program called Chef. Chef manages the provisioning of servers, and is used especially in cloud,VM and multi-server installations to manage all of the server instances. 

Monday, February 24, 2014

Can't Connect: HeidiSQL + VirtualBox + mySQL + Ubuntu

After creating a new mySQL server within an Ubuntu guest OS running on VirtualBox, (how is that for a mouthful?) I wanted to connect to the server with HeidiSQL, my toolbox of choice when dealing with basic SQL operations. I expect that I would have the same problems attempting to connect with similar  client software, such as mySQL Workbench or PHPmyAdmin.

The symptoms of the problem showed up in an error message, generated by HeidiSQL:

Connect Error (2003) Can't connect to MySQL server on '' (10061)  

So,  more or less in order, I did the following:
1. Create a mySQL user with an identical name and password as the Ubuntu user account. To connect to mySQL, you need to have two valid accounts:  a.) a valid Linux user account,  and b.) a valid mySQL user account. On Ubuntu, these need to be same....because you only get one chance to pass through a name and password when logging into a server from HeidiSQL. This is done by logging into the mySQL client on the Ubuntu box, and performing the following:



Exit the mySQL client, and then reload the grant tables, using the mySQL root user and password 
mysqladmin -u root -prootpassword reload 

Note that the both the user names and password are case-sensitive, and the mySQL user's name and password must be identical to the ubuntu user's name and password for Heidi to be able to log in. 

2. In VirtualBox, be sure that the network settings for your network card are for a bridged adapter. What this does is it makes the VM act like another machine on your current network. The VirtualBox default is to create its own network and subnet which are inaccessible from the host machine.  (but see note #1 below) 

3. Once you've taken care of the above items, restart the VM, and log in at the console. You need to edit the mySQL configuration file to reflect the new IP address of the VM.  Log into the VM. Usually the opening screen will show the current IP address,  in the line: 

IP address for eth0:    

If you miss it or need to find it again, just run

ifconfig | grep "inet addr" 
This will probably show at least two lines. The first one is the current IP address.   

inet addr: 192168.219.107 Bcast: Mask:255.255.2550 
inet addr: Mask 

Now that you know the address, you can add it to mySQL configuration file: 

sudo nano /etc/mysql/my.cnf 

Scroll down in this file until you find an entry for the bind address. This will look something like: 
bind-address = 

Change this so it it listens on the address of your VM.  I just comment out the bind-address line, and add the new line: 

#bind-address =
bind-address  =

Restart the machine.
sudo shutdown -r now 

This reloads everything, including the mySQL daemon so that it listens on the new IP address. 

That's it. You should be able to log in with Heidi now.

1. You could probably leave the default networking setting in place, (it is NAT), and the VM will be assigned a NAT address along the lines of  You'd still probably want to edit the mySQL file to reflect this address, and you might also want to make sure that you forward the mySQL port 3306.  I just haven't tested this, so that I can't vouch that it will work.

2. Of course, in a production situation, you'll want to choose a fixed IP address for the network address. This might involve your main router and firewall, so you may need to discuss this with the sys admins.

Anyway, now that I've actually got this thing running...I've got to get back to mySQL!

Friday, February 21, 2014

Tech Friday: Note from the Cranky Sysadmin

1. With VirtualBox chugging along, I've have developed super SysAdmin (systems administrator) powers, and created and destroyed dozens of Linux servers and desktop workstations using both Ubuntu and Red Hat operating systems. 

2. It takes over an hour to format an 8 gigabyte USB thumb drive in Windows 7. A quickformat will take just a couple of seconds. 

3. The Boss took a look at my old Linux books, (copyright 2002) and laughed a mirthless hollow laugh. "Those are really old books".  I said, "yes, they are, but they happen to describe your really old operating systems." 

4. The object of all of this virtual goodness is to upgrade the hardware on a currently production server that is around eight-years old. The problem is that the operating system, mySQL and PHP are equally old. Our idea was to transfer the existing production server, on to the new hardware, and then worry about upgrading, as we expect that some of the PHP will need to be rewritten. The problem is similar to the proverbial "change a tire on a car at 70 miles per hour". To date, I've been working on several separate scenarios:

  • Shut down the server, and do a bit-copy of the old system to the new system 
  • Leave the server running and do a copy to the new system over the network.
  • Create a new empty server with the old versions of the OS (Redhat 7) and mySQL (3.23). Dump the mySQL data off the old server and restore to the new server. 
I'm pretty confident the first option will work, as long as the server can be booted from a CD or USB, and as long as it can be rebooted once we've taken it down. 

As near as I can figure out, the server has two mirrored 40gigabyte drives, which result in a single RAID drive at /dev/hdd.  In Theory... I can do a copy of /dev/hdd to the new hardware and boot.  

[root@hive root]# fdisk -l
Disk /dev/hda: 40.0 GB, 40020664320 bytes255 heads, 63 sectors/track, 4865 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/hda1 * 1 191 1534176 fd Linux raid autodetect
/dev/hda2 192 1211 8193150 fd Linux raid autodetect
/dev/hda3 1212 1721 4096575 fd Linux raid autodetect
/dev/hda4 1722 4865 25254180 f Win95 Ext'd (LBA)
/dev/hda5 1722 1912 1534176 fd Linux raid autodetect
/dev/hda6 1913 1977 522081 fd Linux raid autodetect

Disk /dev/hdc: 40.0 GB, 40020664320 bytes
16 heads, 63 sectors/track, 77545 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Device Boot Start End Blocks Id System
/dev/hdc1 * 1 3045 1534176 fd Linux raid autodetect
/dev/hdc2 3045 19301 8193150 fd Linux raid autodetect
/dev/hdc3 19301 27429 4096575 fd Linux raid autodetect
/dev/hdc4 27429 77536 25254180 f Win95 Ext'd (LBA)
/dev/hdc5 27429 30473 1534176 fd Linux raid autodetect
/dev/hdc6 30473 31509 522081 fd Linux raid autodetect

Disk /dev/hdd: 40.0 GB, 40020664320 bytes64 heads, 32 sectors/track, 38166 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Device Boot Start End Blocks Id System
/dev/hdd1 1 38166 39081968 83 Linux
The crankyness is partly of my own making, and partly, just Linux, I think. For every command there is a separate program with dozens of options making it almost impossible to memorize anything. Since everything I'm doing at this stage doesn't involve routine or repetition,it makes little sense to invest time into doing any scripting, which is, as far as I can tell, the traditional way a Linux guru copes with the command line. Usually after six months of being immersed in a foreign language, it begins to Make Sense. (that worked fine before). 

But Linux remains elusive. That's where Virtualbox comes in; you can rehearse operations and try things out.       

Wednesday, February 12, 2014

VirtualBox: Running Headless

Following up on the previous discussion of running VirtualBox from the command line, the next task is to be able to run a remote VirtualBox session, that is, a virtual machine hosted on a remote computer which may or may not be capable of a graphics display. In this case, the virtual machine is started from the host's command line using the headless parameter

VboxManage startvm RHDesktop --type headless 
This starts the vm, and automatically by default starts the remote desktop server, to allow you to log into the machine using Microsoft's Remote Desktop or any other client that uses the RDP protocol. There are at least two quirks regarding the log in with Remote Desktop:

1. When configuring the port for RDP, in VirtualBox settings,  use a port other than the default RDP 3389.
(I used 5000).

2. When connecting to the Virtual Machine using RDP, use  the the IP address of the host for the IP address.  Note that the IP address used is independent of any IP address that is assigned to the guest operating system, either as a fixed address or through NAT, when bridging.

(click on the images to see them full-sized.)

There is a good YouTube Video showing this with Windows hosts and guests.

With my RedHat guest, there seems to be issues with the mouse accurately tracking on the remote desktop.  It will be interesting to see if this is a major problem or not.

Friday, February 07, 2014

VirtualBox: Working from the command line.

Besides a superb graphical user interface, VirtualBox has a command-line interface which can be run from a command line or terminal session. You have to navigate to the installation directory for VirtualBox. On Windows 7 it is located in
C:\Program Files\Oracle\VirtualBox

Once in the directory, you control the virtual machines by using the vboxmanage command followed by a subcommand, followed by the name of the VM that you want to work with.  

List the currently configured VMs.  
 vboxmanage list vms

"UbServer1" {86742a7d-1188-4b48-b09a-4a0139d14930}
"UbServer2" {30bbee61-7115-4494-91fb-987a1dad74a4}
"UbDesktop" {a83379da-31e4-4064-b4c7-de4dec73195d}
"RHServer71" {bf9515d0-aead-4423-990b-3a6493712dc0}
"RHDesktop" {461c94b9-6743-4d7b-a444-6972586a562b}

This shows that I have 5 configured VMs.... two Ubuntu servers, a Ubuntu desktop, and two RedHat virtual machines.

To find out information about a specific VM:

vboxmanage showvminfo ubServer1

Name:            UbServer1
Groups:          /
Guest OS:        Ubuntu
UUID:            86742a7d-1188-4b48-b09a-4a0139d14930
Config file:     C:\Users\Larry\VirtualBox VMs\UbServer1\UbServer1.vbox
Snapshot folder: C:\Users\Larry\VirtualBox VMs\UbServer1\Snapshots
Log folder:      C:\Users\Larry\VirtualBox VMs\UbServer1\Logs
Hardware UUID:   86742a7d-1188-4b48-b09a-4a0139d14930
Memory size:     2048MB
Page Fusion:     off
VRAM size:       12MB
CPU exec cap:    100%
HPET:            off
Chipset:         piix3
Firmware:        BIOS
Number of CPUs:  1
PAE:             on
Long Mode:       off
Synthetic CPU:   off
CPUID overrides: None
Boot menu mode:  message and menu
Boot Device (1): Floppy
Boot Device (2): DVD
Boot Device (3): HardDisk 

This is the beginning of the listing, which runs about five screens. It includes all of the information shown in the GUI Settings pane, as well as additional information. 

vboxmanage startvm RHDesktop

Waiting for VM "RHDesktop" to power on...
VM "RHDesktop" has been successfully started.

This starts a named virtual machine.
The counterpart for stopping is:

vboxmanage controlvm RHDesktop poweroff

 Full manual is here

Thursday, February 06, 2014

VirtualBox: Getting Started

Having worked with virtualization software since prehistoric times... (yes, Quarterdeck DeskView and QEMM) it only has really worked for me with machines with expanded RAM. We have come a long way. The best recommendation I can give is that if you are hosting on Windows, that you would do well to have 4-8 gigs of RAM. You need to have enough RAM to comfortably run the host operating system as well as the guest operating system(s). This may be less of a problem for older guests, and maybe Linux, without a GUI (we'll see ...).

If you are expecting to be productive on the host machine when running a large guest OS, you need the RAM. For example, on my 4 gig iMac, I don't do much in the way of Mac stuff, when I'm running Windows 7 via Parallels, and I've only allocated 1.5 gigs of memory to the Windows guest... so Windows doesn't perform particularly well either.

On the other hand with my recent Windows 7 box with 8Gigs, I've been going to town hosting multiple Linux versions, as well as Windows 7 and 8 guests. In these cases I'm able to allocate 2-4 gigs to each guest.  

To get started with VirtualBox, you need to go to the website and download the version you want for your operating system. Examples here are from the Windows version. Mac and Linux versions behave similarly. Once installed and started you are shown the VB Manager.

Click to enlarge
The manager shows each virtual machine set up on the host in the left-hand pane, and then anything specific to a guest in the right hand pane. This includes all of the hardware specifications for the virtual hardware, as well as a screen shot of what is happening on the screen of the virtual machine.

The manager currently shows an instance of Red Hat Version 9 (Shrike) booting up. The preview screen is an echo of a full interactive screen which is present in its own window.  Below is an example of a Windows 7 guest running within a Windows 7 host. 

Cick to enlarge: Installing a Windows 7 guest 
Guest operating systems can be installed from physical CD/DVDs, USB drives, or disk images (.iso files). Iso files are easiest, as you don't have to worry about the VirtualBox host using external hardware. 

For example for the Windows 7 installation shown above, I assigned the Windows installation .iso file by clicking on the disk in the upper right-hand corner in the VirtualBox manager. 

Click to enlarge: Choose an disk image for installing an OS

VirtualBox: Create your own cloud with virtual machines

Virtual Machine (VM) software has been around for a long time. Mac users are familiar with Parallels. On the Windows side there have been several to choose from, including one from Microsoft itself which is delivered with some versions of Windows 7, to allow users to run Windows XP programs. VirtualBox is an open source, free program available from Oracle.  It is easy to set up and fun to play with.

But wait, why would I use this?  I can think of three reasons:

1. Run an essential program on one system that doesn't work on another system. A classic use for VM software is to run a different operating system (the guest) on your main computer (the host). Many Mac users run Parallels to be able to run Windows for an essential program that isn't on the Mac (think QuickBooks...for small businesses, or some CAD or 3d imaging programs.)  I run Windows 7 on my iMac and laptop Macbook Pro using Parallels, but they would work fine with VirtualBox).

2. Run an older version of something that you just can't conveniently upgrade. My own motivation for investigating this is to be able to extend the service life of a creaky mySQL database that has connections to a web site. While we're in the process of rewriting the web site, it will still be awhile before we can ditch the old database, and in the meantime, we're having hardware issues. So we're hoping to use VirtualBox to migrate the old server on to a VM running on new hardware.

3. Consolidate servers. Virtual Machines can be used to provide multiple servers on a single hardware server, saving space and energy.      

4. Research, rehearsal, training and testing. Using a VM is a great way to rehearse and train for a hardware upgrade, to try out a new operating system, or to play with (virtual) hardware. Just as NASA astronauts simulate a moon landing, you can easily simulate a five-drive RAID array, and figure out how to set one up without all those pesky wires.

Next: Set up VirtualBox on Windows 7.

Sunday, February 02, 2014

Odds and Sods

Don Lancaster has posted many of his books as free PDF files. My favorite has always been the Incredible Secret Money Machine.

Life seems to go in three year cycles. So every now and then it pays to take an inventory and see if what you are doing and who you are doing it with is what you really want. Are you now handling something for some organization that is being completely thwarted by some petty politics or childish ego games? Is the course you are teaching still relevant? Are you doing what you want with who you want? Are your biases and prejudices hurting you? Do you have., all the facts? What is working and what isn't? What is fun and what is not? Are the hours you are spending on any project paying for themselves or otherwise rewarding you? Are you ignoring obvious new directions? Is a guilt trip getting laid on you? Have one time friends become psychic energy sinks? Are church or family demanding too much of your energy? 

The above paragraph is from the introduction to the 1992 second edition. The first edition came out in 1978. (I note these are going for $150.00 on Amazon).

LogMeIn has dropped their LogMeIn Free service.  For those with current Central or paid Pro accounts, nothing seems to have changed as of yet.

A recently published paper attempts to quantify the attributes of successful KickStarter campaigns. The Language that Gets People To Give.

Chris Oatley, a graphics and character designer has a series of blog posts entitled, Will Your Personal Project Make Money?

The Missing FileMaker ExecuteSQL Reference by Beverly Roth. If you think you need it, you need it. The link is from the excellent FileMaker Hacks web site.

MightyData: Another web site, full of Filemaker hacks.

Working on a migration project to move an older mySQL database to new hardware, and in the process, I'm simulating it as much as possible using VirtualBox. So far the challenges have been learning enough about how VirtualBox works, coupled with mySQL grants and rights, and a fair amount Ubuntu Linux, including software RAID.

Thursday, January 23, 2014

SmartyStreets: Interactive Address Verification

Among the many gifts that the Internet bestows on non-profits, SmartyStreets surely has to be one of them.

Put in an address.....say,  1 Santa Claus Lane, North Pole AK...


and you will get back a listing of the correct postal address, including the zip+4 address,  a map of the location, the time zone of the location, and a slew of information about the address. If it is a commercial address, you may get the company name included.

If you are doing data entry, and want to confirm an address on the fly, you can do this interactively by just bringing up the web page.  SmartyStreets will return the address, corrected and formatted for a postal label. SmartyStreets will also perform minor corrections on an address.  For example, if you put in 10 Oak Street, and the address is really 10 Oak Drive, SmartyStreets, will suggest Oak Drive as a valid street name.
Great stuff!  

Tuesday, January 07, 2014


Having been MIA for much of the fall of 2013, working on mailing lists for a local non-profit. I've learned a couple things: 

FileMaker Pro version 13 has been released. The price has increased, and the paradigm for multi-user deployment has changed in favor of their browser-based workstation client as opposed to the stand-alone client. I'm not sure that it is as economically viable for small deployments as it was in the past, but the Windows version of FileMaker Pro Advanced is still a terrific program for manipulating databases of any kind. About US $325.00 to upgrade or $500 to buy new. For anyone bogged down with a bunch of Excel spreadsheets, FileMaker is really worth a look; you can download a 30-day trial.  Works with Macs, Windows, iPhones, and iPads. and provides a very nice front end to ... 

mySQL  An open-source alternative to SQL-Server.  There is a phpAdmin administrator program which is popular, and may be fine for administration, as opposed to querying, but for a really nice lightweight front-end SQL query generator HeidiSql is freeware, and it works really well (Windows only). Heidi has syntax checking and coloring, maintains a query history, runs batch queries, and does just about anything a normal user needs. 

Lenovo ThinkCenter desktops. A solid Windows 7 workstation, especially when the RAM is increased to 8 gigabytes. Utterly unglamorous, utterly reliable. A great alternative to Dell.  About $600.00 from NewEgg.  

SmartyStreets This web site provides address verification for mailing lists. It allows you to verify a single address at a time as you are entering or editing your lists, or it allows you to submit a full list of almost any number of addresses for verification and correction. It will provide the "plus-4" for zip codes, and will also provide the bar-code numbers if you want to put bar codes on your address labels. There is a lot here, and I'm just getting started with it. Free for non-profits for up to something like 150,000 names. Note it only deals with U.S. addresses. 

GroveStreams is a cloud-based charting a dashboard-display program. I've described this more over on as I interface my Arduino heart-rate monitor to display data on GroveStreams.