The HOTBED Project required a working system to gain research outputs from the second term of the academic year 2001/2, the broad user requirements for which are detailed below. This involved the development from scratch of a bespoke system, as well as the installation of client workstations within the Academy, and the presentation of a prototype for initial feedback.

The aim was for the HOTBED system to be tested informally throughout the final term of 2001/2 with consistent monitoring by HOTBED staff to measure usage as well as usability. This culminated in more formal, structured user feedback sessions at the end of that term. The system was then developed into a second-generation model with increased functionality (see below) – HOTBED v2.0 – which was delivered in September 2002, for integration into the curriculum in the academic year 2002/3.

1. Accessible on the World-Wide-Web (W3)
2. Easily navigable (i.e. predictable to anyone familiar with the W3)
3. Learning and Teaching oriented
4. Resource focussed. The core of the system would be traditional Scottish music sound materials and their associated data.
5. Secure. Outstanding copyright issues necessitated the initial limiting of access to staff and students of the RSAMD and School of Scottish Studies.

Logistics of the placing of client machines around the Academy caused unexpected difficulties throughout the development of the prototype system, and began at this initial planning stage. Room booking procedures, security issues, lack of networking provision all took a great deal of working through with representatives from many different departments. Eventually arrangements were made for the placing of 5 clients (2 in teaching rooms, 2 in practice rooms, and 1 in a designated Scottish music soundlab), each of which would be connected to the staff network for access to the W3.

Given the research-based focus of the HOTBED project, it was clear that the system would have to be adaptable. In addition, looking forward to the exit strategy at project close, maintainability of the system would also be a key concern. With these two issues in mind, and taking cost implications into account, open source technologies were chosen for webserving (Apache), database (MySQL), and middleware (PHP). The rapidly widening use of these technologies and their in-built integration also bolstered their appeal.

The Apple G4 server package was chosen as it offered the hardware required – 500 MHz PowerPC 750, 256 Mb RAM, 60GB storage, and Gigabit Ethernet – along with compatibility with the open source technologies mentioned above out-of-the-box. In addition, it runs OSX Server, which is built on a UNIX core for straightforward, and well-understood system administration and reliability, and Quicktime Streaming Server (QTSS) which would allow us an easily configurable streaming system for the HOTBED resources. The quality of streamed-media was also a factor in opting for Quicktime over competitors such as Real and Windows Media, as it is reported to offer superior sound quality to both. Quicktime Streaming also has no associated hidden costs per stream, and so long term ‘success costs’ would be avoided.

Problems with the installation and with the basic connection of the server to the LAN were minimal. It was configured to pick up its IP, etc using DHCP, and this worked for basic webserving within the firewall. Later problems to do with the firewall impacted the project heavily and are detailed below.

Apache 1.3 was configured to run its own PHP4 module and a site with the name ‘hotbed.ac.uk’ was created. The domain name ‘hotbed.ac.uk’ had been purchased from UKERNA, who were awaiting two delegated nameservers to place the entry in the DNS. Until this was remedied, access to the website and developing system could only be obtained by IP addresses – one for internal use, and another for use outwith the firewall. (An Academy proxy translates outside requests into internal addresses.)

Backup solutions were then considered for server (website and database) and digitisation work. Tape drives were investigated, but were dismissed as being too expensive, as the backup of systems would be weekly rather than daily, and the amount of data is unlikely to exceed the tens of gigabytes in the foreseeable future. (Additionally, no satisfactory tape backup solution was available for Mac OS X Server at the time of purchase.) As a result the newly released Iomega Peerless drive was purchased. It benefits from being portable – allowing for the easy migration of data from encoding machine to server, for example - expandable, and easy to administer. Two cartridges were purchased – one for offsite backup holding.

A freelance web designer was brought in to revamp the look of the website, improve its usability, and to free up resources for overall project management and system design. Access to the new system was linked-to from the front page of the new site restricted with login and passwords, which could then be distributed to staff and students (both at the RSAMD and the School of Scottish Studies) in a controlled manner.

As the HOTBED system would be centred around dynamic web content, the site was designed with this in mind, and dynamic content was added to each page using SHTML and CGI for template includes and ‘last updated information’. After the HOTBED system itself was operational, however, this would change (see below).

Persons are described separately in the database, thereby allowing for the build of a powerful search based on Performers, Arrangers, or Composers. In addition, the database has been created to allow for the description of a Group so that Performers can be described collectively.

Instrument also exists as an entity within HOTBED database and allows the relation of a single Instrument to many Performers and vice versa.

Once the model was finalised and agreed upon, SQL (Structured Query Language) was written to create the database in MySQL. Work on system development was carried-out offsite and tested thoroughly before being ported to the server (using Peerless). A home-based LAN, using a combination of Linux (Redhat 7), Mac OS9, and Windows 2000 (for testing only), was used to write the PHP, Javascript, and SQL required by the system. A Linux machine running Apache, PHP, and MySQL was used as the development server.

During these initial development stages, a six week digitisation period was undertaken by a temporary member of staff who was brought in specifically for the task. The initial digitisation workflow was:

1. Record the source material to the PC using CoolEdit
2. Normalise and top-and-tail the audio material.
3. Record as a .wav file (16 bit 44.1kHz) for archiving to CD.
4. Encode the file to MPEG 1 Layer 3 at 128 kbps using the Fraunhofer codec in CoolEdit.
5. Detail the description (metadata) of the new file in the HOTBED database.

There were a few technical problems with the digitisation/encoding process itself. The first was the sourcing of appropriate technology to play 78 records. A player was found within the Academy, but a stylus had to be specially ordered. The playback of 78s also caused problems, in that they may have been recorded at a speed that was not exactly 78 rpm, thereby altering the pitch of the piece, sometimes radically. This often necessitated pitch change work in CoolEdit.

The one big technical issue we faced was with a persistent click infesting a large batch of materials encoded. Eventually this was identified as noise from the new ethernet connection on the PC. The encoding machine is now a standalone unit with no network connection.

Due to the demands of getting materials into the database as soon as possible, the first section of the HOTBED system to be created was the Admin Area. Pages were written to deliver a metadata form that would enable digitisation staff to use the Web to enter description in the database while digitising the corresponding sound resources, without their needing to know or understand SQL. Each form, upon submission to the HOTBED database, would return the Identifier_ID automatically generated by the database server. This ID would then be used to rename the newly created mp3 sound file so that it could be referenced in a URL query string dynamically generated with PHP pulling the ID from the MySQL database.

Once the first admin forms were completed and the digitisation period begun, the core of the HOTBED system was written, the first part of which was the search engine (search.php). HOTBED staff agreed that this would be a targeted search in V1.0, and six categories were identified for the first version: Performer, Composer, Title, Instrument, Tune Type, and Language. The first three of these searches would be text based searches, while the remainder would pull their options from a controlled list, already described in the HOTBED database. A search page was created, its emphasis on simplicity, with each of the six search criteria presented as highlighted hyperlinks.

PHP functions were then created to generate the SQL appropriate for each of these searches and these were separated into their own ‘includes’ file for easy maintenance and expansion later on. Once the SQL was finalised a results page (results.php) was written, which displayed the SQL results table in chunks of n results, where n = the user preference to be set later on (see below). Next and Previous links were created to allow navigation between the results chunks; the query simply calling the same results page again, using a pointer variable to point to the current position within the results set. This was a tricky process, as it had to allow for first page, last page, and last page with fewer than n results.

The results were displayed as a list of hyperlinks, pulling only the Title of each ArchiveItem from the database. (This was expanded later – see below) Each hyperlink then generated a URL with the appropriate ID tagged onto the query string that would lead to an item page.

The item page (item.php) was then written, consisting of a limited set of metadata in V1.0. It was developed in a modular fashion, to allow the easy insertion of more of the metadata fields later on. It simply pulled each of the desired fields from the database and displayed them in a tabular format, with a link to a Javascript popup for the associated mp3 as the final item in the table.

The Javascript function created a new; custom sized browser window, with no toolbar, controls, etc, and held only an embedded Quicktime player to allow users to listen to the associated sound resource. In the full v1.0 system the Javascript popup was abandoned in favour of an embedded Quicktime player, as it was seen to be superfluous.

As a final stage in the prototype a secure login system was created – once again using PHP and MySQL. User data was added to the HOTBED database, including the following info: login, password, and status. Status would allow us to restrict access to certain areas (such as Admin) further down the line. Each page within the system was then set up with a session management system which would check whether a user had logged-on, and force a logon if the result was negative. In addition, information on the user’s preferences (for number of results per page, etc.) and last login time would be kept.

Once all of this was completed and tested on the Linux Development Server, the MySQL database and PHP pages were ported over to the Mac Delivery Server using Peerless. No problems were encountered here. The MySQL database is stored in backup form as a complete SQL record in flat file text (.txt) format, allowing instant (re)creation of the database. The system worked on OS X Server as it had done on Linux.

The mp3 files were renamed with their appropriate HOTBED Identifier_IDs and the system – from search to item discovery – worked well. The final step before first demonstration involved the integration of the new system into the existing website, with its look and feel. HTML for headings, navigation, etc was copied form the existing website pages and a HOTBED web template was created using PHP, and included at the beginning of each new PHP page. Similarly a footer was created, to allow ‘last updated’ and other information to be generated on the fly. Again, this was done in PHP and included at the close of each new page. (This left us in the strange position of having PHP includes in the HOTBED system and CGI includes on the public website: this would be remedied later when all the includes files – for templates, footers, etc – were rewritten in PHP.)

The prototype was demonstrated to two Scottish Music staff, both of whom responded very favourably. Feedback from this short session was then fed back into a short burst of development to bring the prototype up to V1.0 for the main demonstration to all RSAMD Scottish Music staff.

HOTBED staff brainstorming sessions (in direct response to the feedback from the prototype demo) led to the suggestion of additions to the Prototype system to bring it up to V1.0 status for the official demonstration. The new functionality issues were once again drawn from UNA sessions and comprised the following:

• User Profiling to identify users on the system individually
• Most Recent Searches List
• ‘Favourites Lists’ that would allow users to collect and group discovered items
• A basic intraHOTBED email system that would facilitate the passing of ‘Favourites Lists’ between HOTBED users.
• A ‘Public Lists’ feature that would allow staff to post lists for all HOTBED users to see.

The User Profiling would also facilitate the tracking of system usage further down the line, thereby allowing an analysis of system usage in different areas. It would also mean that copyright concerns could be diminished to an extent, as a secure login would be required to access HOTBED. This involved the creation of a new table in the HOTBED database, with fields to describe a username, password, profile (a textual description of the user – full name, position, etc), status (whether the user is staff, student, HOTBED staff, and so on), and last accessed time. The status field was key to ensuring that only those users with staff status could create public lists, and also allowed the creation of an admin area on the website which could only be accessed by HOTBED staff. (The metadata forms were placed here.)

The most Recent Searches List was constructed by creating another new DB table that comprises a list of search URLs associated with users’ IDs and the text of the search query.

Similarly, the Favourites List is a DB table with fields for name of list, user ID, description, date created, and an access state. The access state allows PHP to find out whether a list is public or private. Another table had to be created due to the many-to-many relationship between users and Favourites Lists, simply marrying user IDs and list IDs.

The message (or ‘email’) system uses yet another MySQL table to store information. The information stored here includes:

• the ID of the user sending the message
• the ID of the addressee
• the message text itself
• the ID of any attached Favourites List
• a sent date (which includes the time)
• and a checked field (which is a flag to allow PHP to decide whether a message has been read yet or not).
• When a user sends a message using HOTBED it is simply stored in the database using the above metadata.

A set of profile pages was created on the site, with the main page displaying the user’s text profile (with an ‘Edit’ link to allow the entry of personal description); a link to the user’s ‘Message Centre’; the time and date when an item in the HOTBED database was last viewed; a box to allow the user to set their preferred number of results per page; the user’s Favourites Lists (with a box to create a new List); and a list of the user’s most recent searches.

Another PHP page was created for the Message Centre (messages.php). It comprises a list of received messages and a link to create a new message. The new message is created with a text box and a drop-down list that contains all the users currently listed in the HOTBED DB. This will quickly become unwieldy however, and a directory of users (detailing username and profile) will have to be created. This will also have the added benefit of allowing groups of users to be addressed at once (all staff for example). Another drop down list pulls all the user’s Favourites Lists from the DB to allow one list to be sent as an attachment. A Javascript alert informs the user of a successful transfer with the name of the message recipient.

The same work process was used for the development of V1.0 as had been used for the Prototype build: offsite coding on the development server, with testing over the LAN. Once it was built and tested, v1.0 was ported over to the delivery server using Peerless. Again, no problems were encountered with the transition.

For the demonstration of v1.0, the server was moved to the demonstration location. Internal networking issues meant that serving was unreliable at best, so the decision was made to demo v1.0 from the localhost. This went smoothly and the response to the new system was very positive indeed.

After the public demonstration of V1.0, project focus shifted to the utilisation of the system within the Academy (as well as minor tweaks and enhancements).

Five HOTBED client machines were installed onsite – two in practice rooms, two in teaching rooms, and one in the Scottish Music soundlab. This proved to be less simple than anticipated, however, as networking had to be provided in the practice rooms and unanticipated security and health and safety issues were highlighted.

A compromise on the security issue (i.e. the potential theft of equipment) was to leave the machines themselves in the rooms, but take the satellite speakers out to be stored elsewhere under HOTBED staff control. This has proven unworkable, however, and a convenient solution has yet to be reached. In addition, the machines are often moved from their tables and left on the floor when not in use, so the purchase of custom cabinets for the machines seems to be the only solution.

Once installed, the clients (running Windows 2000) had the Academy’s login system (Novell) installed and a ‘hotbed’ user was created by Academy IT staff. Each of the machines boots straight into Internet Explorer after successful network login and is taken directly to the HOTBED homepage.

At this point testing of the streaming server revealed that clients within the Academy were not receiving the streamed materials. Further testing revealed that streaming outwith the firewall was possible, but, due to the firewall configuration itself, was not possible to internal clients. As the materials could be delivered using progressive download the introduction of pure streaming was delayed until essential IT reconfiguration work had been completed by Information Services over summer break 2002. As security of copyright was no longer a big concern (progressive download being almost as secure) the only outstanding issue here is the size of the delivered files. Access to the multimedia materials offsite from a standard 56k dialup remained unrealistic until the Quicktime Streaming Server was started.

At this stage, our practice of encoding to mp3 and re-encoding to Quicktime movie was re-examined. While the sound quality remained high, and the sound file size low, the system was not the most efficient. The files were therefore re-encoded to Quicktime Movie format directly from wav using the Qdesign codec in Discreet Cleaner and two alternate streams were provided: one broadband and one narrow.

A public access to the catalogue (i.e. the metadata) held in the HOTBED database was created to allow any user to examine the HOTBED holdings without danger of infringing copyright on the multimedia materials. A public user was created in the HOTBED user DB and the main item page (item.php) was amended to check for this user and prevent the loading of the Quicktime player if appropriate. Additionally, the main search page (search.php) also checks for such a user and removes links to individual profiling items such as admin, lists, and message pages. The login page (access/index.php) was then altered to provide the choice between logging-in as a registered or a public user.

The results page (results.php) was also enhanced following comments from users that more information regarding each item would be beneficial (especially as two or more items could share the same Title). An additional query level was added to the page to pull out and display the Performer and Recorded Date (if known). An outstanding issue with this facility is that of multiple performers, the SQL only pulling out the first (named) Performer associated with each item. One possibility would be to pull out all performers (but this could easily become unwieldy); another would be to have a ‘main performer’ associated with each item (and this would involve a slight reworking of the data structure).

HOTBED v1.0 was therefore a working system that delivered multimedia (currently sound) materials via a searchable database. It also provided messaging, profiling, and organisational tools to enhance learning Scottish Music over the Web.

However, the HOTBED v1.0 was also designed to handle video – the Quicktime Server will deliver it the same was it does the sound materials, and we had purchased a video editing package comprising Adobe Premier. Specially commissioned video was shot for HOTBED v2.0 and the item.php code was rewritten to allow the display of video.

The format field of the database was used to detect whether the item was video, and code was written to ascertain this at the display stage. If a video item was detected the appropriate window would be displayed. Given the flexibility offered by the Quicktime format there were no significant problems in this area.

The search was also revamped and extended for use in HOTBED v2.0:

• A browse facility was added in order to allow users the ability to have a look through the existing items in HOTBED without having to specifically search for an item. This was achieved by using the HOTBED v1.0 search engine to display all existing results in each of the categories used for the targeted search.
• A free text search was also developed to allow a user to specifically target certain phrases or names across all categories in the HOTBED database. This was designed to search across only meaningful categories in the database (i.e. not the fields that contained keys or internal database references). These results are then ordered by the search based on the number of ‘hits’ for each word in the search phrase found in the returned item. Each of the returned items is added to a database table that is created on the fly using the user’s unique id and this table is indexed based on these hits. Once used the table is then destroyed.
• An ‘intelligent search’ was then added to the free text search in order to allow users to search using synonyms. As an instrument, for example, is held in the database as ‘Pipes – Highland’, searching for ‘bagpipes’ would yields no results. We created a synonym table in the HOTBED DB that mapped possible synonyms to their entry in HOTBED. Any admin user can add a synonym and each item can have any number of synonyms. The search was amended so that it checks for existing synonyms and will search using these synonyms as well.

The final major development built into v2.0 was a Multimedia Manipulation Tool (MMT) for looping and marking of an audio/visual resource. Although initial research into this field suggested that access to Quicktime time controls could only be accessed through the Java API, further investigation showed that the API could be accessed through Javascript on most current browsers (with the main exception being IE on the Mac due to licensing difficulties between Apple and Microsoft).

These Java routines facilitated the development of a bespoke program that allows a user to click on a ‘mark’ button which calls the get time routine with subsequent clicks of the created marker calling the set time routine. A loop button then sets the start and end time and switches on the loop facility in the Quicktime player, providing user control of the loop.

In order to avoid duplication of work on the part of the user, the database was again augmented to provide storage for each of the markers a user wishes to save. A new page was written which provides the user with the facility to delete or save markers. This page then calls MySQL routines to write the Quicktime timecode along with the user’s ID and the item ID to the database in order to display the markers even after the user has logged off and back on.

Again, given HOTBED’s use of the Quicktime format, this system works for both audio and video.