Generating Pre-Signed URLs for Amazon S3 with ColdFusion

We've recently been trying to get one of our applications to play nicely with Amazon S3 to deliver output rather than storing it on our production servers. Now ColdFusion 9.0.1 plays nicely with S3 in terms of uploading content to S3 but it doesn't have an answer when the content you want is in a private bucket and you therefore need a signature with an expiry for an authorised user to download.

Barney Boisvert has written his own CFC here which seemed to work a treat... until it didn't. In testing it proved to be inconsistent as to whether AWS would let you in or not; in addition different browsers (e.g. Chrome/IE) seemed to deal with the encoded URLs differently and would tell you your signature was invalid.

So the answer lay in the actual Amazon AWS SDK for Java. Download it and add it to your WEB-INF/cfusion/lib folder. You will also need to acquire the following third party jar files to add into the same place as the SDK is missing them for ColdFusion:

  • httpclient-4.1.1.jar
  • httpcore-4.1.jar
  • jackson-core-asl-1.4.3.jar
  • mail-1.4.3.jar
  • stax-api-1.0.1.jar
  • stax-1.2.0.jar

Restart your CF server for these libraries to be loaded.

So for your code, you'll need a function like this:

view plain print about
1<cffunction name="javaS3Url">
2    <cfargument name="bucket" type="string" required="true" />
3     <cfargument name="objectKey" type="string" required="true" />
4
5    <cfscript>
6    
7    var url = '';
8    var accessKey=
<your access key>;
9    var secretKey=<your secret key>;
10    var thisDate = dateAdd("d",1,now());
11    var clientConfig = createObject('java',    'com.amazonaws.ClientConfiguration').init();
12    var thisProtocol = createObject('java', 'com.amazonaws.Protocol');
13    
14    clientConfig.setProtocol(thisProtocol.HTTP);
15    
16    var awsCred = createobject('java',
17        'com.amazonaws.auth.BasicAWSCredentials').init(
18        accessKey,
19        secretKey        
20        );
21    var s3 = CreateObject(    'java',
22        'com.amazonaws.services.s3.AmazonS3Client'
23        ).init(awsCred, clientConfig);
24        
25    url = s3.generatePresignedUrl(bucket, objectKey, thisDate );
26    
27    return url;
28    </cfscript>
29    
30    </cffunction>

The above code takes your object key, creates an expiry date of 24 hours, configures the AWS config to use HTTP not the default HTTPS, creates an S3 client object with your credentials (called "s3") and then calls the method generatePresignedUrl on the object with your bucket and object key. The function returns a properly signed URL you can now display and use.

The reason for using HTTP not the default HTTPS is that with S3 buckets named with dot notation (e.g. sword.sci-ware.com) the certificate will not match *.s3.amazonaws.com. So unless you have a simple bucket name (e.g. swordsciwarecom) browsers will throw security exceptions at you. Your content is already secured via a signature so you don't really need both.

This last point however took ages to get right due to difficulties in making use of a Java enum (in this case being able to set the protocol to be HTTP). In order to set the protocol to be HTTP you need to create a Protocol object (but not use init()) and pass that object with the suffix .HTTP (which is the enum) to the method on the config object.

So then. ColdFusion. Why?

As Steve Bryant has declared today International How I Got Started In ColdFusion Day here is my piece.

My initial exposure was relatively limited; in 1997 I was doing a bit of research into providing access to IBM AS/400s running JBA software via the Internet and came across Allaire Cold Fusion. I was immediately able to connect to the DB2 database and pull back some data into a web page, which at the time seemed pretty magical.

I didn't come to use it in anger until 2000 when the company I was then working for decided to get into Content Management Systems. It started a partnership with a small company called Goss Interactive who had written a CMS in ColdFusion and as the presales person who was going to be showing it I had to learn all about it. The CMS was OK but I found it more fun to write internal applications for the previously non-existent Intranet using ColdFusion.

After a spot of travelling and data analysis type jobs I finally rocked up at Science Warehouse in 2005 as an Implementation Consultant. Although not hired as a programmer one of the first things I did was to introduce ColdFusion as part of the application stack by writing a business intelligence suite ("e-reports") which is widely used by our customers.

Now we have several applications and the main public website running ColdFusion happily alongside the main J2EE application, which means we can get new functionality out there a lot quicker than with Java alone.

Faster, Better, Sooner

This is a full version of a blog post which appears here: http://blog.sci-ware.com/2011/07/12/part-i-agile-software-development-rapid-results-for-customers/

In software, fads, trends and buzzwords come and go. However one concept which has persisted and grown since it first appeared is that of "Agile Software Development". This is not a specific methodology, more of a philosophy and grouping of methodologies that in 2001 were defined as "Agile" in the Agile Manifesto.

The Manifesto for Agile Software Development states:

"We are uncovering better ways of developing software by doing it and helping others do it.

Through this work we have come to value:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan
  • That is, while there is value in the items on the right, we value the items on the left more."

    Lightweight software development methodologies had been promoted since the 1990s (such as Scrum, Extreme Programming and Dynamic Systems Development Method (DSDM)) as a reaction to the traditional highly regulated and micromanaged waterfall method of writing software. By "waterfall" we mean a sequential, highly-defined process of gathering requirements, producing a design, coding, testing and maintenance whereby once one step is completed you progress to the next and you don't go back to the start.

    The reason for the prevalence of this model in the past was that this is how physical products and structures are built; a late change to the design of a bridge or a car is very costly to implement so it is imperative to get it right at the early stages. As software development in the 1970s became more structured and the role of computers in society became more important this well-understood engineering methodology was imported.

    However, as we have seen in recent years with the expensive failure of public and private IT projects this approach depends on meticulous and correct design and analysis up front before any work is done (or at least work beyond a prototype.) Despite the large numbers of Project Managers, Committees and Consultants engaged on a project to ensure it did what was intended and didn't overrun in time and cost, inevitably the high complexity of these projects meant they always did.

    If new requirements emerge or existing ones change over the course of the project, or mistakes are found, or technology changes (quite possible in a large IT project) then what is finally built may meet the specification but won't meet the ultimate needs of the user or customer by the time it sees the light of day.

    These issues will only become more prevalent. It is my view that software development has some of the characteristics of a craft as well as of engineering. Any developer or designer will tell you that there is a creative element to writing code; there are often many solutions to a given requirement but some are more efficient, some more elegant, some will allow greater future expansion, some are quicker to write and so on.

    It is this creativity which makes metrics such as lines of code per day meaningless. It is the final deliverable piece of functioning software which counts, and for that functionality to be fit for purpose.

    A craftsperson cannot be told what to do every minute of the day, nor mandated to do things in a certain way. As the 'talent', they can be taught techniques, 'house style', process etc but you have to let them get on with the job. Management then becomes direction and decision making, not standing over people and obsessing about micro metrics.

    This brings us back to Agile. The lightweight methods are now considered together as "Agile Development" because they have certain common themes which mitigate against some of the problems traditional methods have:

    1. Rapid and frequent delivery of useful and working functionality - in weeks not months
    2. Welcoming of and adapting to changing requirements - as this means the final product will be more useful
    3. Sustainable, consistent pace of development - to avoid developer burnout
    4. Daily co-operation and communication between developers and the business
    5. Trust developers and designers to know and do their jobs

    Science Warehouse and Agile

    The Agile methodology we use to develop software is called Scrum. There are a number of different roles and 'artifacts' involved in Scrum, such as:

    Scrum Team: A Scrum Team is usually around 7-12 people strong and contains analysts, developers, designers and testers - anyone you need to build your product. As far as Scrum is concerned you are a Team Member and can do whatever you can to fulfill the needs of customers; you are not restricted by job title.

    Product Owner: In the world of Scrum this is the person who represents the voice of the customer. For us this is Jon Moody, our Product & Technology Director. It is Jon's job to determine from the requests made by customers and the strategic direction of our software what the priorities ought to be.

    ScrumMaster: This is part of my role - it's my job to keep the various parts of Scrum working and to facilitate the work of the Team.

    Daily Scrum: a 10-15 minute stand up meeting at the start of every day where each team member updates the others on three things: What they have done since the last Daily Scrum, what they plan to do today and whether anything is impeding them in doing so. This isn't a meeting to update management but to let each other know what they are working on and encourage skills transfer

    User Stories: Prior to allocating any work, our Business Analyst Jack Towse takes change requests and turns them into what we call User Stories. These are discrete pieces of functionality described from the point of view of the end user. For example, we might have a request for a change to the Price Approval module. This is analysed and created as a Story along the lines of "As a Procurement Department User I want to be able to see all price updates pending my approval in one page", with the appropriate inputs and outputs that developers need to know about.

    Sprints: Currently we release software every 6 weeks. This is more frequent than similar companies in our market (who typically release every 3 months) but it means that we can deliver the most important requirements for customers more often without a huge wait. Our releases are split into two internal iterations (in Scrum parlance these are called Sprints), with user stories requiring greater QA resources being done in the first iteration to give more time in the second for changes. As a consequence, if we misunderstand the requirements or they change significantly in some way then we've only lost 6 weeks of work and not potentially 6 months or more.

    Other meetings at the start and end of releases are designed to review what has just happened and take learning points (both good and bad) to take into the next release, as well as plan what is coming up.

    Day-to-Day practices

    What these Scrum concepts mean in practice for us is that we trust our development team to do their jobs. As I have the role of ScrumMaster within Science Warehouse it's my job to remove any impediments in the way of the team and to facilitate the various meetings that make up the Scrum method. In addition, as Development Manager I bear in mind the principles of the self-organising team to let the experts do their jobs but provide support and direction where required.

    At different times in the development cycle team members will use Agile practices such as programming pairs to solve a particular issue or to draw on complementary skills (such as a designer and developer working at the same PC).

    Regular releases mean that we strive to automate as many processes as we can. Recently we have introduced Continuous Integration, which along with helping to create and distribute the final build of code can be used to continuously test the latest code for defects.

    Conclusion

    Through the use of Agile software development methods Science Warehouse are able to respond quicker to evolving requirements and deliver the software that our customers need the most more frequently. We produce documentation that is needed by people not processes, saving time. Through our incremental and iterative approach we drive out errors and flaws much faster than more traditional processes would allow.

    Finally the use of Agile means that our team are more productive and feel more trusted in their work as professionals. Being given a structure and responsibility has improved the self-confidence and job satisfaction of our development team; this along with our relaxed office environment makes Science Warehouse a great place to work and develop your career, and in turn we get fantastic results from the team.

    What's most important is that the Agile approach means we are constantly reviewing our practices and that improvements come from the team: what's good about what we produce as a company comes from them.

Mitre - My Tweets, Retweeted

Mitre icon

Mitre, the first mobile application from us here at The Northern Union, is now available on the iTunes app store. Download it.

Mitre lets you see who has retweeted any of the last 20 Twitter tweets any user has made. I used the oft-retweeted user@FrancisUrquhart to test as most of my own tweets tend not to be retweeted!

I've got a limited number of free download credits if anyone fancies testing and reviewing it for me. Get in touch if you do via email: james@northern-union.com

Mitre Support

Mitre is now available on the Apple App Store. Go get it!

If you need support for this application email us: support@northern-union.com

Thanks!

I've written a BlogCFC pod...

I'm now trying to put a little bit of time aside to devote to this website. I'm actually more interested in doing the development of it than writing content but that's because I've got some free ColdFusion hosting and I'm going to use it!

You can see the first fruits of my efforts with my Tweets pod on the right. I'm reusing the Twitter CFC I wrote for the Science Warehouse homepage and using my Twitter name of @jamesahull instead of the @sciware one.

The tricky bit was to understand how to use variables in the application. The way I did it was to add this to Application.cfm:

view plain print about
1<cfset application.tweets = createObject("component", "org.northern.getTweets").retrieve()>

Then create a tweets.cfm page in the pods directory and use this code:

view plain print about
1<cfmodule template="../../tags/podlayout.cfm" title="@jamesahull">
2
3    <cfoutput>
4    <p class="center">
5    <cfset tweets = application.tweets>
6    <cfset x = structCount(tweets)>
7        <cfloop from='1' to='#x#' index='i'>
8            <div class="tweets">
9                #tweets.text[i]#
10                <div class="tweettime">
11                    #tweets.time[i]#
12                </div>
13            </div>
14        </cfloop>
15    </p>
16    </cfoutput>
17
18</cfmodule>

Pretty easy, although if you want me to post the actual code to parse Twitter then leave a comment.

Next stop, actually doing some design work on the default blog skin....

Foul Brood

The baddest tune of the lot, from Rotator.

Can't see the player? Follow these links: Foul Brood by Northern Union

St Kilda

A cormorant infested rock or a formerly-seedy fashionable Melbourne suburb? YOU DECIDE

Can't see the player? Follow these links: St Kilda by Northern Union

Mornington

Here's my Soundcloud recording of Mornington.

Can't see the player? Follow these links: Mornington by Northern Union