March 3, 2023
Writing an Analytics Server using Vapor Part 5 - The Database
In the previous post, I finished my work on the route to add a UserEvent to the database. However, I don't yet have a database.
Choosing a Database
Vapor offers a pacakge called Fluent for interacting with a database. It's an ORM (Object–relational mapping) framework that lets me write code that will interact with ANY database, and then choose one of several databases to work with.
Eventually, I will want something performant and efficient that can store values over time. From what I can tell, PostgreSQL is the preferred database for tabular data in the Vapor community. So when I deploy this, I'll want to use PostgreSQL.
For now though, what I need is something that stores data in a tabular format and is fast and efficient. I don't actually care about persisting data yet, since I am still writing my data layer and things are likely to change a lot before I'm done. So an in-memory database is probably all I need for now. Vapor offers the ability to set up a SQLite in-memory database (docs).
I change my configure.swift file to use an in-memory SQLite database:
public func configure(_ app: Application) throws {
app.databases.use(.sqlite(.memory), as: .sqlite)
try routes(app)
}
The documentation recommends setting up a migration and then automigrating the database. I'll do that soon, but first I want to think some more about how things are going to be saved.
Deciding on a Persistence Strategy
It's time to think again about the model layer. So far, I have a single UserEvent struct that is being sent back and forth successfully via the endpoint. Fluent provides a Model protocol for storing a type in a database (docs). It expects a value type. So I have the option of switching UserEvent to a value type or writing a separate type to be stored in the database.
It makes more sense to me to write a separate Model type and translate between the two within UserEventController. I can decode the request into a UserEvent, then build a UserEventRecord from it and save that to the database. I can then translate that UserEventRecord back into a UserEvent to return it in the response. This may sound like more work, but it establishes a clear separation of concerns. The UserEvent is for interfacing with the route, while the UserEventRecord is for interacting with the database.
Reconsidering UserEvent
Since I'll be using a separate UserEventRecord for storing the event in the database, I should reconsider the layout of UserEvent. I had included an id property in UserEvent, thinking that it would be needed by the database. But if UserEvent itself is just a transport object, it probably doesn't need an id field. So I remove the id field:
struct UserEvent: Content, Equatable {
let userID: UUID
let flag: Bool
let timestamp: InternalDate
enum Action: String, Codable {
case start, pause, stop
}
let action: Action
init(action: Action, userID: UUID, flag: Bool = false) {
self.userID = userID
self.timestamp = InternalDate(Date())
self.flag = flag
self.action = action
}
}
There was a test that explicitly checked to make sure that a 400 is sent if the json doesn't include an id.
func test_post_responds_with_400_if_not_given_id_in_payload() async throws {
var propertiesWithMissingValues = exampleValidUserEventProperties
propertiesWithMissingValues.removeValue(forKey: "id")
let data = try JSONSerialization.data(withJSONObject: propertiesWithMissingValues)
try await testPOST(ByteBuffer(data: data)) { response in
XCTAssertEqual(response.status, .badRequest)
}
}
Obviously, this is no longer needed and can be removed.
Also, exampleValidUserEventProperties no longer needs to include an id key-value pair
private var exampleValidUserEventProperties: [String:Any] {
[
// "id": UUID().uuidString,
"userID": UUID().uuidString,
"timestamp": Date().timeIntervalSinceReferenceDate.rounded(),
"flag": true,
"action": UserEvent.Action.start.rawValue
]
}
UserEventRecord Model Object
The UserEventRecord has all the same properties as UserEvent. I will need to be able to create a UserEventRecord from a UserEvent, and I'll need to be able to convert a UserEventRecord back into a UserEvent.
I abhor magic strings, and I will need to use the same strings to define both the keys in the UserEventRecord and the fields in the database migration, so I start by adding some custom constants to FieldKey.
extension FieldKey {
static var timestamp: FieldKey { #function }
static var flag: FieldKey { #function }
static var action: FieldKey { #function }
static var userID: FieldKey { #function }
}
Then I create the UserEventRecord class and conform it to Model.
The timestamp property requires some special thought. I'll store it in the database as a Double representing timeIntervalSinceReferenceDate.
The action property isn't a problem at all in this situation, because Fluent provides a special @Enum property wrapper for string-backed enums.
For now, userID will just be stored as a UUID. I may decide to do something more relational with it later.
final class UserEventRecord: Model {
static var usereventrecords: String { #function }
static let schema = usereventrecords
@ID(key: .id)
var id: UUID?
@Field(key: .userID)
var userID: UUID
@Field(key: .timestamp)
var timestamp: Double
@Field(key: .flag)
var flag: Bool
@Enum(key: .action)
var action: UserEvent.Action
init() { }
}
Then I add a convenience init for creating a UserEventRecord directly from a UserEvent:
init(id: UUID? = nil, _ userEvent: UserEvent) {
self.id = id
// take the timestamp from the userEvent,
// don't use a Fluent @Timestamp
// because we want the time that the event happened
// not the time that the record was created
self.timestamp = userEvent.timestamp.value.timeIntervalSinceReferenceDate
self.userID = userEvent.userID
self.flag = userEvent.flag
self.action = userEvent.action
}
and I add a computed property for getting a UserEvent out of a UserEventRecord:
var userEvent: UserEvent {
return UserEvent(
date: Date(timeIntervalSinceReferenceDate: timestamp),
action: action,
userID: userID,
flag: flag)
}
Adding the UserEvent to the database
Adding the UserEvent to the database involves creating a UserEventRecord from the UserEvent in the request, saving it in the database, and returning the UserEvent associated with the new UserEventRecord. It's very straightforward:
func create(req: Request) async throws -> UserEvent {
let event = try req.content.decode(UserEvent.self)
let record = UserEventRecord(event)
try await record.create(on: req.db)
return record.userEvent
}
But when I try to do this, I get multiple failed tests.
The most glaring one happens on
func test_post_responds_with_200() async throws {
which gives the following message:
test_post_responds_with_200(): XCTAssertEqual failed: ("500 Internal Server Error") is not equal to ("200 OK")
A look through the console reveals many errors, but the first one is:
2023-03-02T11:46:27-0500 warning codes.vapor.request : request-id=0E87539B-A731-4D66-B558-238BA24AC2B4 [] error: no such table: usereventrecords
So Fluent is complaining because the database hasn't been set up yet.
Set up a Migration
Fluent uses Migrations to set up and revert the database (docs). In order to store the UserEventRecords in the database, I need to set up a table for the UserEventRecords to be stored. Luckily, it's a simple matter of building a SchemeBuilder object, which is dead simple if you've ever done any declarative programming.
struct CreateUserEventRecords: AsyncMigration {
func prepare(on database: FluentKit.Database) async throws {
try await database.schema(UserEventRecord.usereventrecords)
.id()
.field(.timestamp, .double)
.field(.userID, .uuid)
.field(.flag, .bool)
.field(.action, .string)
.create()
}
func revert(on database: FluentKit.Database) async throws {}
}
I use the same FieldKey constants that I set up before, and I simply tell the SchemeBuilder what type to expect for each field and then call its create() method.
The revert() method may come in handy after I start persisting the database to disk, but for now it is just a noop.
I update the configure.swift file to use the CreateUserEventRecords migration and then tell the database to automigrate.
public func configure(_ app: Application) throws {
app.databases.use(.sqlite(.memory), as: .sqlite)
app.migrations.add(CreateUserEventRecords())
try app.autoMigrate().wait()
// register routes
try routes(app)
}
Now I hit command-U and all tests pass.
Add a New Endpoint to List UserEvents
But wait a minute. I think I'm adding UserEvents to the database, but I'm not yet able to read them.
We're going to need another endpoint, one that responds to a GET, that lists all the userevents. "uservents/list" would be the ideal name.
So I add a static var on UserEventController
static var listPath: String { [userevents, list].joined(separator: "/") }
And I add a test:
func test_get_list_returns_200() throws {
try sut.test(.GET, UserEventController.listPath) { response in
XCTAssertEqual(response.status, .ok)
}
}
I add a new route method in UserEventController:
func list(request: Request) async throws -> [UserEvent] {
[]
}
and I add the route in UserEventController.boot():
group.get(.list, use: list)
All the tests pass, but that's not so exciting because It's not doing anything yet.
I add another test to make sure that before any UserEvents are added, the endpoint returns an empty array.
func test_get_list_returns_empty_array_if_no_userevents_have_been_created() throws {
try sut.test(.GET, UserEventController.listPath) { response in
let received = try JSONDecoder().decode([UserEvent].self, from: response.body)
XCTAssertEqual(received, [])
}
}
This one fails. You see, the middleware I added in the previous post is checking the header and noticing that there's no verbose header key, so then it returns an empty string and a 200 status code.
My middleware strategy was a little too ambitious. It shouldn't modify endpoints that get data, only endpoints that are posting data. So I modify the UserEventController.boot() again, splitting the routes up into two groups:
func boot(routes: Vapor.RoutesBuilder) throws {
let getroutes = routes
.grouped(.constant(Self.userevents))
getroutes.get(.list, use: list)
let postroutes = getroutes
.grouped(HeaderCheckingMiddleware(key: Self.verbose, value: Self.verboseTrue))
postroutes.post(use: add)
}
Now the tests pass again, but I'm not yet retrieving the UserEvents from the database.
I add another test that sends a POST to the base endpoint and then tries to send a GET to the list endpoint:
func test_get_list_returns_userevent_that_has_been_added() throws {
let sent = UserEvent(action: .start, userID: exampleUserID)
let expected = [sent]
_ = try sut.sendRequest(.POST, UserEventController.userevents, headers: defaultHeaders, body: sent.toByteBuffer())
try sut.test(.GET, UserEventController.listPath) { response in
let received = try JSONDecoder().decode([UserEvent].self, from: response.body)
XCTAssertEqual(received, expected)
}
}
It of course fails.
I modify UserEventController.list() to access the database:
func list(request: Request) async throws -> [UserEvent] {
try await UserEventRecord.query(on: request.db)
.all()
.map(\.userEvent)
}
and the test passes.
I add another test that sends several POST requests and expects to get them all back in a list request:
func test_get_list_returns_all_userevent_that_have_been_added() throws {
let sent = (0..<Int.random(in: 3..<20)).map { _ in
UserEvent(date: Date().addingTimeInterval(.random(in: 60...3600)), action: .allCases.randomElement()!, userID: exampleUserID, flag: .random())
}
try sent.forEach {
_ = try sut.sendRequest(.POST, UserEventController.userevents, headers: defaultHeaders, body: $0.toByteBuffer())
}
try sut.test(.GET, UserEventController.listPath) { response in
let received = try JSONDecoder().decode([UserEvent].self, from: response.body)
// use Sets since order doesn't matter
XCTAssertEqual(Set(received), Set(sent))
}
}
I have to do some protocol conformance to make sure that things that have to be Hashable are and things that have to be CaseIterable are, but in the end everything compiles and all tests pass.
Querying By Date
So far, the userevents/list endpoint isn't very useful. For my needs, I want to know things like "How many user events happened in the last day?" Having the server just send me back every UserEvent that has ever happened will be overkill for a need like this, and it will lead to a lot of network traffic very quickly. I need to be able to get back just the UserEvents that happened in a given time range.
Probably the best way to do this is with queries. I can include a query in the URL that gives a start date and end date, and UserEventController.list() can then filter the UserEvents to just those that fit between those two dates. Since I'm already passing dates back and forth using timeIntervalSinceReferenceDate, I can continue doing that for this query.
So I add a new test:
func test_get_list_returns_all_userevent_that_fit_in_date_range() throws {
let oneDay: TimeInterval = 24*3600
let now = Date()
let startOfDay = Calendar.current.startOfDay(for: now)
let endOfDay = Calendar.current.startOfDay(for: now.addingTimeInterval(oneDay))
let beforeToday = now.addingTimeInterval(-oneDay)
let afterToday = now.addingTimeInterval(oneDay)
let eventBeforeToday = UserEvent(date: beforeToday, action: .allCases.randomElement()!, userID: exampleUserID, flag: .random())
let eventNow = UserEvent(date: now, action: .allCases.randomElement()!, userID: exampleUserID, flag: .random())
let eventAfterToday = UserEvent(date: afterToday, action: .allCases.randomElement()!, userID: exampleUserID, flag: .random())
try [ eventBeforeToday, eventNow, eventAfterToday]
.forEach {
_ = try sut.sendRequest(.POST, UserEventController.userevents, headers: defaultHeaders, body: $0.toByteBuffer())
}
let path = UserEventController.listPath +
"?" +
"startDate=\(startOfDay.timeIntervalSinceReferenceDate)" +
"&" +
"endDate=\(endOfDay.timeIntervalSinceReferenceDate)"
try sut.test(.GET, path) { response in
let received = try JSONDecoder().decode([UserEvent].self, from: response.body)
XCTAssertEqual(received, [eventNow])
}
}
Yes, this is a very verbose test. I'll be refactoring it very soon. But first let's get it working.
Vapor will let me decode a struct from a query string (see the docs on Content), so I create a struct that can be created from a query with a startDate and endDate using my InternalDate struct:
struct DateRangeQuery: Content {
var startDate: InternalDate
var endDate: InternalDate
// convenience methods for filtering
var start: Double { startDate.value.timeIntervalSinceReferenceDate }
var end: Double { endDate.value.timeIntervalSinceReferenceDate }
}
Then, I start by simply returning an empty array if the DateRangeQuery can be created:
func list(request: Request) async throws -> [UserEvent] {
if let dateRange = try? request.query.decode(DateRangeQuery.self) {
return []
}
return try await UserEventRecord.query(on: request.db)
.all()
.map(\.userEvent)
}
This still causes the test to fail, but it does so in a different way:
test_get_list_returns_all_userevent_that_fit_in_date_range(): XCTAssertEqual failed: ("[]") is not equal to ("[App.UserEvent(userID: F152BB12-6663-44B8-801A-1FFF132E30B8, flag: true, timestamp: App.InternalDate(value: 2023-03-03 15:32:54 +0000), action: App.UserEvent.Action.pause)]")
So now it's just a question of filtering the UserEventRecords to only give us the ones we want. Fluent of course offers a robust QueryBuilder for deciding what items to return from a query (docs). It takes some exploring to get the syntax right (I probably should have started with a simpler field, like the flag field), but I eventually get the filter working:
func list(request: Request) async throws -> [UserEvent] {
let query = UserEventRecord.query(on: request.db)
if let dateRange = try? request.query.decode(DateRangeQuery.self) {
return try await query
.filter(\.$timestamp >= dateRange.startDate.value.timeIntervalSinceReferenceDate)
.filter(\.$timestamp <= dateRange.endDate.value.timeIntervalSinceReferenceDate)
.all()
.map(\.userEvent)
}
return try await query
.all()
.map(\.userEvent)
}
And the test passes.
Refactoring the Test
That test was extremely verbose, and it included some code that I may end up wanting to use elsewehere, so let's see if I can make it more succinct.
I'll start off with the random UserEvents. Yes, it makes sense to have random UserEvents for a test like this, but the repetitive calls to random() everywhere just make the test hard to read. So I factor that out into an extension on UserEvent:
extension UserEvent {
static func random(at date: Date) -> UserEvent {
UserEvent(date: date, action: .allCases.randomElement()!, userID: .generateRandom(), flag: .random())
}
and I use that method to create the UserEvents in the test:
let eventBeforeToday = UserEvent.random(at: beforeToday)
let eventNow = UserEvent.random(at: now)
let eventAfterToday = UserEvent.random(at: afterToday)
I can also use it in a test from before:
let sent = (0..<Int.random(in: 3..<20)).map { _ in
UserEvent.random(at: Date().addingTimeInterval(.random(in: 60...3600)))
}
Now that path that I created. It's pretty verbose, and I can imagine wanting to write other tests that require query stirngs, so I write a helper method:
func pathString(_ path: String, adding queries: [(String, String)]) -> String {
guard queries.count > 0 else { return path }
return path + "?" + queries.map { "\($0)=\($1)" }.joined(separator: "&")
}
and rewrite the code to read:
let path = pathString(UserEventController.listPath, adding: [
("startDate", String(startOfDay.timeIntervalSinceReferenceDate)),
("endDate", String(endOfDay.timeIntervalSinceReferenceDate))
])
but that's not that much better, so I add a convenience method just for creating a valid path to the userevents/list endpoint with or without queries:
func listPath(startDate: Date? = nil, endDate: Date? = nil) -> String {
var queries = [(String, String)]()
if let startDate {
queries.append(("startDate", String(startDate.timeIntervalSinceReferenceDate)))
}
if let endDate {
queries.append(("endDate", String(endDate.timeIntervalSinceReferenceDate)))
}
return pathString(UserEventController.listPath, adding: queries)
}
and I use it in the test:
try sut.test(.GET, listPath(startDate: startOfDay, endDate: endOfDay)) { response in
I also use it in any other test against the uservents/list endpoint:
func test_get_list_returns_200() throws {
try sut.test(.GET, listPath()) { response in
XCTAssertEqual(response.status, .ok)
}
}
...
I can anticipate wanting to be able to query against the action or flag of the UserEvent, so I go on and add those to the convenience method:
func listPath(startDate: Date? = nil,
endDate: Date? = nil,
userID: UUID? = nil,
action: UserEvent.Action? = nil,
flag: Bool? = nil) -> String {
var queries = [(String, String)]()
if let startDate {
queries.append((UserEventController.startDate, String(startDate.timeIntervalSinceReferenceDate)))
}
if let endDate {
queries.append((UserEventController.endDate, String(endDate.timeIntervalSinceReferenceDate)))
}
if let userID {
queries.append((UserEventController.userID, userID.uuidString))
}
if let action {
queries.append((UserEventController.action, action.rawValue))
}
if let flag {
queries.append((UserEventController.flag, String(flag)))
}
// shuffle the queries to ensure that the server is robust about how it handles queries in any order
return pathString(UserEventController.listPath, adding: queries.shuffled())
}
Finally, the code to send the UserEvents is pretty verbose. I create a helper method that sends the userEvent to the expected endpoint:
func post(_ userEvents: [UserEvent]) throws {
try userEvents.forEach {
_ = try sut.sendRequest(.POST, UserEventController.userevents, headers: defaultHeaders, body: $0.toByteBuffer())
}
}
Which gives me a final test that looks like:
func test_get_list_returns_all_userevent_that_fit_in_date_range() throws {
let oneDay: TimeInterval = 24*3600
let now = Date()
let eventToday = UserEvent.random(at: now)
try post([
UserEvent.random(at: now.addingTimeInterval(-oneDay)),
eventToday,
UserEvent.random(at: now.addingTimeInterval(oneDay))
])
let startOfDay = Calendar.current.startOfDay(for: now)
let endOfDay = Calendar.current.startOfDay(for: now.addingTimeInterval(oneDay))
try sut.test(.GET, listPath(startDate: startOfDay, endDate: endOfDay)) { response in
let received = try JSONDecoder().decode([UserEvent].self, from: response.body)
XCTAssertEqual(received, [eventToday])
}
}
Other tests also send UserEvents to that endpoint in order to set up the databse, so I add a helper for the helper:
func post(_ userEvent: UserEvent) throws {
try post([userEvent])
}
and I replace the other calls to sut.sendRequest in the other tests.
Testing an Obvious Edge Case
What happens if the user passes in an end date that is before the start date? You would expect to just get an empty array, but there should be a test for that:
func test_get_list_returns_empty_array_if_endDate_precedes_startDate() throws {
let now = Date()
let eventToday = UserEvent.random(at: now)
try post(eventToday)
let startOfDay = Calendar.current.startOfDay(for: now)
let endOfDay = Calendar.current.startOfDay(for: now.addingTimeInterval(.oneDay))
try sut.test(.GET, listPath(startDate: endOfDay, endDate: startOfDay)) { response in
let received = try JSONDecoder().decode([UserEvent].self, from: response.body)
XCTAssertEqual(received, [])
}
}
and this test passes as expected.
Testing a Less Obvious Edge Case
But what if only a start date is given? You would think that you'd get all UserEvents whose timestamp happens after the startDate, but that's not what happens with this test:
func test_get_list_returns_empty_array_if_not_given_endDate() throws {
let now = Date()
let eventToday = UserEvent.random(at: now)
let eventTomorrow = UserEvent.random(at: now.addingTimeInterval(.oneDay))
try post([
UserEvent.random(at: now.addingTimeInterval(-.oneDay)),
eventToday,
eventTomorrow
])
let startOfDay = Calendar.current.startOfDay(for: now)
try sut.test(.GET, listPath(startDate: startOfDay)) { response in
let received = try JSONDecoder().decode([UserEvent].self, from: response.body)
XCTAssertEqual(Set(received), Set([eventToday, eventTomorrow]))
XCTAssertEqual(received.count, 2)
}
}
Instead, all 3 UserEvents are sent back.
This is because the DateRangeQuery expects both a start date and an end date, and if it doesn't find both, then UserEventController.list right now just defaults to returning every UserEventRecord.
Honestly, for my needs, I would say that requesting a startDate but not an endDate, or vice-versa, should be considered an error. It would probably make more sense to return a 400 (Bad Request) status code in this case.
So I write a pair of tests that expect to get a 400 response if only a start date or only an end date appear in the query:
func test_get_list_returns_400_if_given_startDate_but_not_given_endDate() throws {
try sut.test(.GET, listPath(startDate: Date())) { response in
XCTAssertEqual(response.status, .badRequest)
}
}
func test_get_list_returns_400_if_given_endDate_but_not_given_startDate() throws {
try sut.test(.GET, listPath(endDate: Date())) { response in
XCTAssertEqual(response.status, .badRequest)
}
}
and I modify UserEventController.list() to respond with a 400 status code if it can't parse the DateRangeRequest:
func list(request: Request) async throws -> [UserEvent] {
let query = UserEventRecord.query(on: request.db)
if let dateRange = try? request.query.decode(DateRangeQuery.self) {
return try await query
.filter(\.$timestamp >= dateRange.startDate.value.timeIntervalSinceReferenceDate)
.filter(\.$timestamp <= dateRange.endDate.value.timeIntervalSinceReferenceDate)
.all()
.map(\.userEvent)
}
else if request.url.query?.isEmpty == false {
throw Abort(.badRequest)
}
return try await query
.all()
.map(\.userEvent)
}
and the tests pass.
In fact, ANY query that contains unexpected keys should cause a 400 status code now. So I should write a test to ensure that this always remains the case
func test_get_list_returns_400_if_query_contains_unexpected_keys() throws {
let path = pathString(UserEventController.listPath, adding: [("foo", "bar")])
try sut.test(.GET, path) { response in
XCTAssertEqual(response.status, .badRequest)
}
}
This may seem a bit strict, but since I intend to be the only one using this server to retrieve data, it makes sense to be very strict about how the requests are formatted.
Refactoring the List Route
Now let's take another look at the route:
func list(request: Request) async throws -> [UserEvent] {
let query = UserEventRecord.query(on: request.db)
if let dateRange = try? request.query.decode(DateRangeQuery.self) {
return try await query
.filter(\.$timestamp >= dateRange.startDate.value.timeIntervalSinceReferenceDate)
.filter(\.$timestamp <= dateRange.endDate.value.timeIntervalSinceReferenceDate)
.all()
.map(\.userEvent)
}
else if request.url.query?.isEmpty == false {
throw Abort(.badRequest)
}
return try await query
.all()
.map(\.userEvent)
}
This logic is kind of convoluted, and if I wanted to respond to other query keys, it could get much worse.
Luckily, query is a Fluent QueryBuilder type, so I can rewrite the route to be much more logical.
So I adjust the logic just slightly to only ever have one query that is always returned at the end (unless there's an error):
func list(request: Request) async throws -> [UserEvent] {
var query = UserEventRecord.query(on: request.db)
if let dateRange = try? request.query.decode(DateRangeQuery.self) {
query = query
.filter(\.$timestamp >= dateRange.startDate.value.timeIntervalSinceReferenceDate)
.filter(\.$timestamp <= dateRange.endDate.value.timeIntervalSinceReferenceDate)
}
else if request.url.query?.isEmpty == false {
throw Abort(.badRequest)
}
return try await query
.all()
.map(\.userEvent)
}
I can then add a method on DateRangeQuery to filter a query:
func filter(_ query: QueryBuilder<UserEventRecord>) -> QueryBuilder<UserEventRecord> {
query
.filter(\.$timestamp >= startDate.value.timeIntervalSinceReferenceDate)
.filter(\.$timestamp <= endDate.value.timeIntervalSinceReferenceDate)
}
and now the route looks much cleaner:
func list(request: Request) async throws -> [UserEvent] {
var query = UserEventRecord.query(on: request.db)
if let dateRange = try? request.query.decode(DateRangeQuery.self) {
query = dateRange.filter(query)
}
else if request.url.query?.isEmpty == false {
throw Abort(.badRequest)
}
return try await query
.all()
.map(\.userEvent)
}
Querying By Action
I may also want to query for just those UserEvents that contain a given action. I can write a test for this.
func test_get_list_returns_all_userevents_that_match_action_requested() throws {
let sent = (0..<Int.random(in: 3..<20)).map { _ in
UserEvent.random(at: Date().addingTimeInterval(.random(in: 60...3600)))
}
let expected = sent.filter { $0.action == .pause }
try post(sent)
try sut.test(.GET, listPath(action: .pause)) { response in
let received = try JSONDecoder().decode([UserEvent].self, from: response.body)
XCTAssertEqual(Set(received), Set(expected))
}
}
and create another struct to extract the action:
struct ActionQuery: Content {
let action: UserEvent.Action
func filter(_ query: QueryBuilder<UserEventRecord>) -> QueryBuilder<UserEventRecord> {
query.filter(\.$action == action)
}
}
Then it's just another if statement in the route:
func list(request: Request) async throws -> [UserEvent] {
var query = UserEventRecord.query(on: request.db)
if let dateRange = try? request.query.decode(DateRangeQuery.self) {
query = dateRange.filter(query)
}
else if let actionQuery = try? request.query.decode(ActionQuery.self) {
query = actionQuery.filter(query)
}
else if request.url.query?.isEmpty == false {
throw Abort(.badRequest)
}
return try await query
.all()
.map(\.userEvent)
}
But what if I try querying for only UserEvents that were created in a certain date range and that also match a given action:
func test_get_list_returns_all_userevents_that_match_action_requested_and_fit_within_date_range() throws {
let now = Date()
let dateRange: ClosedRange<TimeInterval> = -.oneDay ... .oneDay
let sent = (0..<Int.random(in: 3..<20)).map { _ in
UserEvent.random(at: now.addingTimeInterval(.random(in: dateRange)))
}
let startOfDay = Calendar.current.startOfDay(for: now)
let endOfDay = Calendar.current.startOfDay(for: now.addingTimeInterval(.oneDay))
let expected = sent.filter {
$0.action == .pause &&
$0.timestamp.value >= startOfDay &&
$0.timestamp.value <= endOfDay
}
try post(sent)
try sut.test(.GET, listPath(startDate: startOfDay, endDate: endOfDay, action: .pause)) { response in
let received = try JSONDecoder().decode([UserEvent].self, from: response.body)
XCTAssertEqual(Set(received), Set(expected))
}
}
Well, every UserEvent that is created in the date range is returned. The action filter is never applied.
So I rewrite the route again:
func list(request: Request) async throws -> [UserEvent] {
var query = UserEventRecord.query(on: request.db)
var queryWasFound = false
if let dateRange = try? request.query.decode(DateRangeQuery.self) {
query = dateRange.filter(query)
queryWasFound = true
}
if let actionQuery = try? request.query.decode(ActionQuery.self) {
query = actionQuery.filter(query)
queryWasFound = true
}
if !queryWasFound && request.url.query?.isEmpty == false {
throw Abort(.badRequest)
}
return try await query
.all()
.map(\.userEvent)
}
and all tests pass.
Querying by UserID
Something similar can be done to query by UserID.
First, create a test:
func test_get_list_returns_all_userevents_that_match_userID_requested() throws {
let sent = (0..<Int.random(in: 3..<20)).map { _ in
UserEvent.random(at: Date().addingTimeInterval(.random(in: 60...3600)))
}
let someEvent = sent.randomElement()!
let filteredUserID = someEvent.userID
let expected = sent.filter { $0.userID == filteredUserID }
try post(sent)
try sut.test(.GET, listPath(userID: filteredUserID)) { response in
let received = try JSONDecoder().decode([UserEvent].self, from: response.body)
XCTAssertEqual(Set(received), Set(expected))
XCTAssert(received.count > 0) // make sure SOMETHING is returned
}
}
Next, write a custom query struct that conforms to Content:
struct UserIDQuery: Content {
let userID: UUID
func filter(_ query: QueryBuilder<UserEventRecord>) -> QueryBuilder<UserEventRecord> {
query.filter(\.$userID == userID)
}
}
Finally, test against the UserIDQuery in the request's query:
if let userIDQuery = try? request.query.decode(UserIDQuery.self) {
query = userIDQuery.filter(query)
queryWasFound = true
}
Querying by Flag
You would think that querying for the flag property would be just as simple.
First, create a test:
func test_get_list_returns_all_userevents_that_match_flag_requested() throws {
let sent = (0..<Int.random(in: 3..<20)).map { _ in
UserEvent.random(at: Date().addingTimeInterval(.random(in: 60...3600)))
}
let expected = sent.filter { $0.flag == false }
try post(sent)
try sut.test(.GET, listPath(flag: false)) { response in
let received = try JSONDecoder().decode([UserEvent].self, from: response.body)
XCTAssertEqual(Set(received), Set(expected))
}
}
Next, write a custom query struct that conforms to Content:
struct FlagQuery: Content {
let flag: Bool
func filter(_ query: QueryBuilder<UserEventRecord>) -> QueryBuilder<UserEventRecord> {
query.filter(\.$flag == flag)
}
}
Finally, test against the FlagQuery in the request's query:
if let flagQuery = try? request.query.decode(FlagQuery.self) {
query = flagQuery.filter(query)
queryWasFound = true
}
And doing this DOES get the new test to pass, but unfortunately, it causes several other tests to fail.
You see, the FlagQuery is created when there's an explicit "flag" key in the query string, but if there's no "flag" key in the query string then it's created with a default value of false. This is not what we want.
So I make the flag property of FlagQuery optional and also make the filter() method return an optional only if flag exists:
struct FlagQuery: Content {
let flag: Bool?
func filter(_ query: QueryBuilder<UserEventRecord>) -> QueryBuilder<UserEventRecord>? {
guard let flag = flag else { return nil }
return query.filter(\.$flag == flag)
}
}
and I alter the call in the route approriately:
if let flagQuery = try? request.query.decode(FlagQuery.self),
let q = flagQuery.filter(query) {
query = q
queryWasFound = true
}
and all tests pass.
A Stress test for list
Finally, lets add a test that adds a whole bunch of UserEvents, then creates a query to find just one of them.
func test_get_list_stress_test() throws {
let now = Date()
let dateRange: ClosedRange<TimeInterval> = -.oneDay ... .oneDay
let sent = (0..<300).map { _ in
UserEvent.random(at: now.addingTimeInterval(.random(in: dateRange)))
}
let startOfDay = Calendar.current.startOfDay(for: now)
let endOfDay = Calendar.current.startOfDay(for: now.addingTimeInterval(.oneDay))
let happeningToday = sent.filter {
$0.timestamp.value >= startOfDay &&
$0.timestamp.value <= endOfDay
}
let expected = happeningToday.randomElement()!
try post(sent)
let path = listPath(startDate: startOfDay,
endDate: endOfDay,
userID: expected.userID,
action: expected.action,
flag: expected.flag)
try sut.test(.GET, path) { response in
let received = try JSONDecoder().decode([UserEvent].self, from: response.body)
XCTAssert(received.contains(expected))
}
}
Summary
And with that, I'm saving UserEvents to the database and retrieving them with a whole range of queries. Next, it might make sense to try to get some value out of this userID that I've been passing back and forth.
Posts in this Series:
- Part 1: Intro
- Part 2: Setting up the Development Environment
- Part 3: Setting up UserEvent
- Part 4: Dynamic Output
- Part 5: The Database
- Part 6: Retrieving Info About Users
- Part 7: Writing a Client
- Part 8: Prepopulating the Database
- Part 9: Filtering Requests
- server project on github